timofejmalinin 3 years ago
parent e5f14dee47
commit fe060b0547

@ -0,0 +1,18 @@
CONTAINERS_NAME=rsrv
# The name of the image to use for the containers
DJANGO_PORT=8000
DEBUG=False
# The django container settings and enviroment
POSTGRES_DB=rsrv_db
POSTGRES_HOST=db
POSTGRES_PORT=5435
POSTGRES_USER=rsrv_user
POSTGRES_PASSWORD=rsrv_pass
POSTGRES_VOLUME_PATH=./pg_data
POSTGRES_HOST_AUTH_METHOD=trust
POSTGRES_VOLUME_BACKUP_PATH=./pg_dumps
# The database container settings and enviroment
MARTIN_PORT=3000
# The martin container settings and enviroment
DOMAIN=social.ru.com
# The frontend and nginx container settings and enviroment

136
.gitignore vendored

@ -0,0 +1,136 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
.idea/
cells1/
cells_final/
cells_final1/
schools_final.xlsx
certs/
pg_data/
.env.dev
.env.prod
.env.prod_busines
static/
media/
data/
nginx/nginx.conf
nginx/nginx.dev.conf
nginx/nginx.conf.prod_business
nginx/nginx.conf.prod
docker-compose.dev.yml
pg_dumps/
django_static/
django_media/

@ -0,0 +1,16 @@
FROM python:3.8
RUN apt-get update &&\
apt-get install -y binutils libproj-dev gdal-bin
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/

@ -1,92 +1,79 @@
# Postamates # Пилот подсистема рекомендательных сервисов
### Инструкция по развёртыванию проекта:
- Установите [docker](https://docs.docker.com/engine/install/ubuntu/)
## Getting started - Установите [docker compose](https://docs.docker.com/compose/install/)
- Положите файлы из репозитория в папку:
To make it easy for you to get started with GitLab, here's a list of recommended next steps. - через GIT:
```bash
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)! git clone git@gitlab.com:endwork_today/dit.git
cd dit
## Add your files git submodule update --init
mkdir pg_dumps
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files sudo chown -R 999:999 pg_dumps
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
``` ```
cd existing_repo - или архивом (должно быть два файла: dit.zip и dit_frontend.zip)
git remote add origin https://gitlab.com/leaders2022/postamates.git ```bash
git branch -M main unzip dit.zip
git push -uf origin main mv dit-main dit
unzip dit_frontend -d dit
cd dit
mv dit_frontend-main/*(D) dit_frontend
rm -rf dit_frontend-main
mkdir pg_dumps
sudo chown -R 999:999 pg_dumps
```
- создайте файл ```dit/.env``` на основе файла ```dit/.env.like```
- создайте файл ```dit/nginx/nginx.conf``` на основе файла ```dit/nginx/nginx.conf.like```
- добавьте сертификаты SSL для домена frontend и для домена backend в папку ```dit/nginx/certs```
- пример создания SSL сертификатов для домена: [LE для NGINX](https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/)
- пример структуры папки ```dit/nginx``` для файла ```dit/nginx/nginx.conf.like```
```
├── certs
│ ├── front
│ │ ├── README
│ │ ├── cert.pem
│ │ ├── chain.pem
│ │ ├── fullchain.pem
│ │ └── privkey.pem
│ ├── options-ssl-nginx.conf
│ └── ssl-dhparams.pem
├── nginx.conf
└── nginx.conf.like
```
- Запустите проект:
```sudo docker-compose up --build```
- Теперь проект доступен по адресу:
[https://REACT_APP_DOMAIN/](https://REACT_APP_DOMAIN/)
- После запуска проекта создайте администратора доступа:
```sudo docker exec -it business_spatiality_django python manage.py createsuperuser```
- Система администрирования проекта по адресу (используйте логин и пароль администратора доступа):
[https://REACT_APP_DOMAIN/admin/](https://REACT_APP_DOMAIN/admin/)
### Команды для обновления проекта:
- Обновите проект и перезапустите его:
```bash
git pull
git submodule foreach git merge origin main
git submodule foreach git pull origin main
sudo docker stop business_spatiality_martin
sudo docker rm business_spatiality_martin
sudo docker-compose up --build
``` ```
## Integrate with your tools ### Команды для бекапирования данных
- Создать бекап базы данных:
- [ ] [Set up project integrations](https://gitlab.com/leaders2022/postamates/-/settings/integrations) ```bash
sudo docker exec -it business_spatiality_pgbackups bash /backup.sh
## Collaborate with your team ```
- Очистить базу данных:
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/) ```bash
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html) sudo docker exec -it business_spatiality_django python manage.py flush
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically) ```
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/) - Восстановление из бекапа:
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html) ```bash
cat pg_dumps/last/business_spatiality_db-latest.sql | sudo docker exec -i business_spatiality_db psql -U business_spatiality_user -p 5435 -d business_spatiality_db -W
## Test and Deploy ```
- Настройка периодических бекапов:
Use the built-in continuous integration in GitLab. Настройка, которая отвечает за это SCHEDULE: почитать о параметрах можно [тут](http://godoc.org/github.com/robfig/cron#hdr-Predefined_schedules)
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
# Editing this README
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
## License
For open source projects, say how it is licensed.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.

@ -0,0 +1,87 @@
version: '3.5'
x-postgres-variables: &postgres-variables
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
POSTGRES_HOST_AUTH_METHOD: "${POSTGRES_HOST_AUTH_METHOD}"
x-django-variables: &django-variables
PYTHONUNBUFFERED: 1
DJANGO_PORT: "${DJANGO_PORT}"
DEBUG: "${DEBUG}"
x-frontend-variables: &frontend-variables
DOMAIN: "${DOMAIN}"
REACT_APP_DOMAIN_URL: "https://${DOMAIN}/"
x-martin-variables: &martin-variables
MARTIN_PORT: "${MARTIN_PORT}"
services:
django:
container_name: ${CONTAINERS_NAME}_django
build: .
command: >
sh -c "python manage.py migrate &&
python manage.py collectstatic --noinput &&
python manage.py runserver 0.0.0.0:${DJANGO_PORT}"
environment:
<<: *postgres-variables
<<: *django-variables
volumes:
- .:/code
- ./django_static/:/code/django_static/
- ./media/:/code/media/
ports:
- "${DJANGO_PORT}:${DJANGO_PORT}"
expose:
- "${DJANGO_PORT}"
restart: always
depends_on:
- db
db:
container_name: ${CONTAINERS_NAME}_db
image: mdillon/postgis
environment:
<<: *postgres-variables
ports:
- "${POSTGRES_PORT}:${POSTGRES_PORT}"
expose:
- "${POSTGRES_PORT}"
volumes:
- ${POSTGRES_VOLUME_PATH}:/var/lib/postgresql/data
command: -p ${POSTGRES_PORT}
martin:
container_name: ${CONTAINERS_NAME}_martin
image: urbica/martin
ports:
- "${MARTIN_PORT}:3000"
environment:
<<: *martin-variables
WATCH_MODE: "true"
DATABASE_URL: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:${POSTGRES_PORT}/${POSTGRES_DB}"
depends_on:
- db
- django
restart: always
frontend:
container_name: ${CONTAINERS_NAME}_frontend
build: dit_frontend
volumes:
- ./build/:/usr/src/dit_frontend/build/
command:
sh -c "yarn build"
environment:
<<: *frontend-variables
depends_on:
- db
- martin
- django

@ -0,0 +1,120 @@
[
{
"model": "service.config",
"pk": 1,
"fields": {
"name": "Point",
"data": {
"id": {
"label": "ID",
"legend": "ID",
"help": "ID"
},
"point": {
"label": "Геометрия",
"legend": "Геометрия",
"help": "Геометрия"
},
"name": {
"label": "Название",
"legend": "Название",
"help": "Название"
},
"adress": {
"label": "Адрес",
"legend": "Адрес",
"help": "Адрес"
},
"district": {
"label": "Район",
"legend": "Район",
"help": "Район"
},
"link": {
"label": "link",
"legend": "link",
"help": "link"
},
"area": {
"label": "Площадь",
"legend": "Площадь",
"help": "Площадь"
},
"price": {
"label": "Цена",
"legend": "Цена",
"help": "Цена"
}
}
}
},
{
"model": "service.config",
"pk": 2,
"fields": {
"name": "Polygon",
"data": {
"id": {
"label": "ID",
"legend": "ID",
"help": "ID"
},
"geometry": {
"label": "Геометрия",
"legend": "Геометрия",
"help": "Геометрия"
},
"cellid": {
"label": "ID клетки",
"legend": "ID клетки",
"help": "ID клетки"
},
"home": {
"label": "Дом",
"legend": "Дом",
"help": "Дом"
},
"job": {
"label": "Работа",
"legend": "Работа",
"help": "Работа"
},
"horeca": {
"label": "Рестораны",
"legend": "Рестораны",
"help": "Рестораны"
},
"brand": {
"label": "Бренды",
"legend": "Бренды",
"help": "Бренды"
},
"univer": {
"label": "Университеты",
"legend": "Университеты",
"help": "Университеты"
},
"hotel": {
"label": "Отели",
"legend": "Отели",
"help": "Отели"
},
"TC": {
"label": "ТЦ",
"legend": "ТЦ",
"help": "ТЦ"
},
"BC": {
"label": "БЦ",
"legend": "БЦ",
"help": "БЦ"
},
"metro": {
"label": "Метро",
"legend": "Метро",
"help": "Метро"
}
}
}
}
]

@ -0,0 +1,22 @@
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'postamates.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()

@ -0,0 +1,7 @@
import os
from django.core.asgi import get_asgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'postamates.settings')
application = get_asgi_application()

@ -0,0 +1,143 @@
"""
Django settings for HACK2 project.
Generated by 'django-admin startproject' using Django 3.2.8.
For more information on this file, see
https://docs.djangoproject.com/en/3.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.2/ref/settings/
"""
from pathlib import Path
import os
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-5czma@e7b(e4v+c*@bkknj(*em%@x52jizednhy6lye)_@ox4@'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = os.getenv('DEBUG', False) == 'True'
ALLOWED_HOSTS = ['*']
# Application definition
INSTALLED_APPS = [
'corsheaders',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'service.apps.ServiceConfig',
'rest_framework',
'django_json_widget',
'django.contrib.gis',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'postamates.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / 'templates']
,
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'postamates.wsgi.application'
# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': os.getenv('POSTGRES_DB', 'postgres'),
'USER': os.getenv('POSTGRES_USER', 'postgres'),
'PASSWORD': os.getenv('POSTGRES_PASSWORD', 'postgres'),
'HOST': os.getenv('POSTGRES_HOST', 'localhost'),
'PORT': os.getenv('POSTGRES_PORT', 5432),
}
}
# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/3.2/topics/i18n/
LANGUAGE_CODE = 'ru'
TIME_ZONE = 'Europe/Moscow'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/
STATIC_URL = '/django_static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'django_static')
MEDIA_URL = '/django_media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'django_media')
# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
CORS_ORIGIN_ALLOW_ALL = True # If this is used then `CORS_ORIGIN_WHITELIST` will not have any effect
CORS_ALLOW_CREDENTIALS = True
CORS_ORIGIN_ALLOW = True

@ -0,0 +1,11 @@
from django.contrib import admin
from django.urls import path, include
from django.conf.urls.static import static
from django.conf import settings
urlpatterns = [
path('admin/', admin.site.urls),
path('api/', include('service.urls')),
]
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

@ -0,0 +1,7 @@
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'postamates.settings')
application = get_wsgi_application()

@ -0,0 +1,12 @@
Django==3.2
djangorestframework==3.11.1
psycopg2-binary==2.9.3
pandas==1.4.2
xlrd==1.2.0
tqdm==4.64.0
gunicorn==19.9.0
django-cors-headers==3.12.0
pyshp==2.3.0
matplotlib==3.5.2
openpyxl==3.0.10
django-json-widget

@ -0,0 +1,12 @@
from django.contrib import admin
from service.models import Polygon, Point, Config
from django.db import models
from django_json_widget.widgets import JSONEditorWidget
# admin.site.register(Polygon)
# admin.site.register(Point)
@admin.register(Config)
class YourModelAdmin(admin.ModelAdmin):
formfield_overrides = {
models.JSONField: {'widget': JSONEditorWidget},
}

@ -0,0 +1,6 @@
from django.apps import AppConfig
class ServiceConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'service'

@ -0,0 +1,49 @@
from django.core.management.base import BaseCommand
import json
from service import models
def create_fixture(name, file_path, merge):
if merge:
with open(file_path, 'r') as f:
old_data = json.load(f)
max_pk = max([x['pk'] for x in old_data])
else:
max_pk = 0
model = getattr(models, name)
model_default_data = {
f.name: {
"label": f.verbose_name,
"legend": f.verbose_name,
"help": f.verbose_name
}
for f in model._meta.fields
}
data = {
"model": "service.config",
"pk": max_pk + 1,
"fields": {
"name": f"{name}",
"data": model_default_data
}
}
with open(file_path, 'w') as f:
if merge:
old_data.append(data)
json.dump(old_data, f, ensure_ascii=False, indent=4)
else:
json.dump([data], f, ensure_ascii=False, indent=4)
class Command(BaseCommand):
requires_system_checks = False
def add_arguments(self, parser):
parser.add_argument('-n', '--name', type=str)
parser.add_argument('-f', '--file', type=str)
parser.add_argument('-m', '--merge', type=bool, default=False)
def handle(self, name, file, merge, *args, **options):
create_fixture(name, file, merge)

@ -0,0 +1,33 @@
from django.core.management.base import BaseCommand
from service.models import Point
from django.contrib.gis.geos import Point as GeoPoint
from pandas import read_excel
from tqdm import tqdm
def import_points(file_path):
Point.objects.all().delete()
point_models = []
df = read_excel(file_path)
df = df.fillna(0)
df_rows = list(df.iterrows())
for id_, i in tqdm(df_rows):
data = i.to_dict()
lat, lng = data.pop('lat'),data.pop('lng')
geometry = GeoPoint(lng, lat, srid=4326)
try:
point_models.append(Point(point=geometry, **data))
except Exception as e:
print(e)
pass
Point.objects.bulk_create(point_models, batch_size=10000)
class Command(BaseCommand):
requires_system_checks = False
def add_arguments(self, parser):
parser.add_argument('-f', '--file', type=str)
def handle(self, file, *args, **options):
import_points(file)

@ -0,0 +1,28 @@
from django.core.management.base import BaseCommand
from service.models import Polygon
import shapefile
from django.contrib.gis.geos import Polygon as GeoPolygon
from tqdm import tqdm
def import_poly(file_path):
Polygon.objects.all().delete()
poly_models = []
shape = shapefile.Reader(file_path)
shape_records = list(shape.shapeRecords())
for i in tqdm(shape_records):
polygon = i.shape.__geo_interface__
data = i.__geo_interface__['properties']
geom = GeoPolygon(polygon['coordinates'][0], srid=4326)
poly_models.append(Polygon(geometry=geom, **data))
Polygon.objects.bulk_create(poly_models, batch_size=10000)
class Command(BaseCommand):
requires_system_checks = False
def add_arguments(self, parser):
parser.add_argument('-f', '--file', type=str)
def handle(self, file, *args, **options):
import_poly(file)

@ -0,0 +1,44 @@
# Generated by Django 3.2 on 2022-07-09 16:01
import django.contrib.gis.db.models.fields
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='Point',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('point', django.contrib.gis.db.models.fields.PointField(srid=4326)),
('name', models.CharField(blank=True, max_length=256, null=True)),
('adress', models.CharField(blank=True, max_length=512, null=True)),
('district', models.CharField(blank=True, max_length=512, null=True)),
('area', models.FloatField()),
('price', models.IntegerField()),
],
),
migrations.CreateModel(
name='Poly',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('geometry', django.contrib.gis.db.models.fields.PolygonField(srid=4326)),
('cellid', models.IntegerField()),
('home', models.IntegerField()),
('job', models.IntegerField()),
('horeca', models.IntegerField()),
('brand', models.IntegerField()),
('univer', models.IntegerField()),
('hotel', models.IntegerField()),
('TC', models.IntegerField()),
('BC', models.IntegerField()),
('metro', models.FloatField()),
],
),
]

@ -0,0 +1,18 @@
# Generated by Django 3.2 on 2022-07-09 16:05
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('service', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='point',
name='link',
field=models.URLField(blank=True, null=True),
),
]

@ -0,0 +1,18 @@
# Generated by Django 3.2 on 2022-07-09 16:17
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('service', '0002_point_link'),
]
operations = [
migrations.AlterField(
model_name='point',
name='price',
field=models.BigIntegerField(),
),
]

@ -0,0 +1,64 @@
# Generated by Django 3.2 on 2022-07-09 17:04
import django.contrib.gis.db.models.fields
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('service', '0003_alter_point_price'),
]
operations = [
migrations.CreateModel(
name='Polygon',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('geometry', django.contrib.gis.db.models.fields.PolygonField(srid=4326, verbose_name='Геометрия')),
('cellid', models.IntegerField(verbose_name='ID клетки')),
('home', models.IntegerField(verbose_name='Дом')),
('job', models.IntegerField(verbose_name='Работа')),
('horeca', models.IntegerField(verbose_name='Рестораны')),
('brand', models.IntegerField(verbose_name='Бренды')),
('univer', models.IntegerField(verbose_name='Университеты')),
('hotel', models.IntegerField(verbose_name='Отели')),
('TC', models.IntegerField(verbose_name='ТЦ')),
('BC', models.IntegerField(verbose_name='БЦ')),
('metro', models.FloatField(verbose_name='Метро')),
],
),
migrations.DeleteModel(
name='Poly',
),
migrations.AlterField(
model_name='point',
name='adress',
field=models.CharField(blank=True, max_length=512, null=True, verbose_name='Адрес'),
),
migrations.AlterField(
model_name='point',
name='area',
field=models.FloatField(verbose_name='Площадь'),
),
migrations.AlterField(
model_name='point',
name='district',
field=models.CharField(blank=True, max_length=512, null=True, verbose_name='Район'),
),
migrations.AlterField(
model_name='point',
name='name',
field=models.CharField(blank=True, max_length=256, null=True, verbose_name='Название'),
),
migrations.AlterField(
model_name='point',
name='point',
field=django.contrib.gis.db.models.fields.PointField(srid=4326, verbose_name='Геометрия'),
),
migrations.AlterField(
model_name='point',
name='price',
field=models.BigIntegerField(verbose_name='Цена'),
),
]

@ -0,0 +1,25 @@
# Generated by Django 3.2 on 2022-07-11 09:03
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('service', '0004_auto_20220709_2004'),
]
operations = [
migrations.CreateModel(
name='Config',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=256, unique=True, verbose_name='Название модели')),
('data', models.JSONField(verbose_name='Данные')),
],
options={
'verbose_name': 'Конфигурация',
'verbose_name_plural': 'Конфигурация',
},
),
]

@ -0,0 +1,38 @@
from django.db import models
from django.contrib.gis.db import models as gis_models
class Polygon(models.Model):
geometry = gis_models.PolygonField(verbose_name='Геометрия')
cellid = models.IntegerField(verbose_name='ID клетки')
home = models.IntegerField(verbose_name='Дом')
job = models.IntegerField(verbose_name='Работа')
horeca = models.IntegerField(verbose_name='Рестораны')
brand = models.IntegerField(verbose_name='Бренды')
univer = models.IntegerField(verbose_name='Университеты')
hotel = models.IntegerField(verbose_name='Отели')
TC = models.IntegerField(verbose_name='ТЦ')
BC = models.IntegerField(verbose_name='БЦ')
metro = models.FloatField(verbose_name='Метро')
class Point(models.Model):
point = gis_models.PointField(verbose_name='Геометрия')
name = models.CharField(max_length=256, blank=True, null=True, verbose_name='Название')
adress = models.CharField(max_length=512, blank=True, null=True, verbose_name='Адрес')
district = models.CharField(max_length=512, blank=True, null=True, verbose_name='Район')
link = models.URLField(blank=True, null=True)
area = models.FloatField(verbose_name='Площадь')
price = models.BigIntegerField(verbose_name='Цена')
class Config(models.Model):
name = models.CharField(max_length=256, unique=True, verbose_name='Название модели')
data = models.JSONField(verbose_name='Данные')
def __str__(self):
return self.name
class Meta:
verbose_name = 'Конфигурация'
verbose_name_plural = 'Конфигурация'

@ -0,0 +1,17 @@
from rest_framework import serializers
from . import models
from service.utils import get_model_column_names
class PolygonSerializer(serializers.ModelSerializer):
class Meta:
model = models.Polygon
fields = get_model_column_names(model, [])
class PointSerializer(serializers.ModelSerializer):
class Meta:
model = models.Point
fields = get_model_column_names(model, [])

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

@ -0,0 +1,16 @@
from django.urls import path
from django.conf.urls import url
from rest_framework import routers
from rest_framework.authtoken import views as rf_views
from . import views
app_name = 'HACK2'
router = routers.DefaultRouter()
router.register('polygons', views.PolygonViewSet, basename='polygons')
router.register('points', views.PointViewSet, basename='points')
urlpatterns = router.urls
urlpatterns += [
]

@ -0,0 +1,2 @@
def get_model_column_names(model, exclude_fields_names):
return [x.column for x in model._meta.get_fields() if str(x.column) not in exclude_fields_names]

@ -0,0 +1,130 @@
import datetime
from service import serializers
from service import models
from rest_framework import viewsets
from rest_framework.permissions import AllowAny
from django.core.serializers import serialize
from rest_framework.response import Response
import json
from django.core.cache import cache
from rest_framework import status
from rest_framework.decorators import action
from django.db.models import Max, Min
from django.conf import settings
from service.management.commands.create_poly import import_poly
from service.management.commands.create_points import import_points
from rest_framework.parsers import MultiPartParser
import os
from django.core.files.storage import default_storage
from django.core.files.base import ContentFile
from service.utils import get_model_column_names
from django.http import HttpResponseRedirect
from django.contrib import messages
def get_min_max_filters(model, exclude_fields_names):
d = {}
l = []
fields = get_model_column_names(model, exclude_fields_names)
for i in fields:
l.append(Min(i))
l.append(Max(i))
qs = model.objects.aggregate(*l)
for i in fields:
d[i] = [qs[f'{i}__min'], qs[f'{i}__max']]
return d
def get_full_filters(model, exclude_fields_names):
d = {}
fields = get_model_column_names(model, exclude_fields_names)
values = model.objects.defer('pk').values(*fields)
for n, i in enumerate(fields):
d[i] = sorted([v[i] for v in values])
return d
def get_model_naming(model):
return models.Config.objects.get(name=f'{model.__name__}').data
class PolygonViewSet(viewsets.ReadOnlyModelViewSet):
serializer_class = serializers.PolygonSerializer
permission_classes = [AllowAny]
parser_classes = (MultiPartParser,)
@action(detail=False, methods=['get'])
def filters(self, request):
d = cache.get('polygon_filters')
if settings.DEBUG:
d = None
if d is None:
d = get_full_filters(models.Polygon, ['id', 'geometry'])
cache.set('polygon_filters', d, 60 * 60 * 24)
return Response(d, status=status.HTTP_200_OK)
@action(detail=False, methods=['get'])
def naming(self, request):
return Response(get_model_naming(models.Polygon), status=status.HTTP_200_OK)
@action(detail=False, methods=['post'])
def file_import(self, request):
try:
file = request.FILES['file'].file
path = default_storage.save(f'cells/cells_{datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}.zip', ContentFile(file.read()))
import_poly(os.path.join(settings.MEDIA_ROOT, path))
cache.delete('polygon_filters')
messages.add_message(request, messages.INFO, 'Данные успешно импортированы')
except Exception as e:
messages.add_message(request, messages.ERROR, f'Ошибка импорта: {e}')
return HttpResponseRedirect('/admin/')
class PointViewSet(viewsets.ReadOnlyModelViewSet):
serializer_class = serializers.PointSerializer
permission_classes = [AllowAny]
queryset = models.Point.objects.all()
def list(self, request, *args, **kwargs):
queryset = self.get_queryset()
d = cache.get('points')
if settings.DEBUG:
d = None
if d is None:
d = json.loads(serialize('geojson', queryset,
geometry_field='point',
fields=get_model_column_names(models.Point,['point'])
))
cache.set('points', d, 60 * 60 * 24)
resp = Response(d)
resp["Access-Control-Allow-Origin"] = '*'
resp["Access-Control-Allow-Methods"] = 'GET,PUT, OPTIONS'
resp["Access-Control-Max-Age"] = '1000'
resp["Access-Control-Allow-Headers"] = 'X-Requested-With, Content-Type'
return resp
@action(detail=False, methods=['get'])
def filters(self, request):
resp = Response(get_min_max_filters(models.Point, ['id', 'point', 'is2025']))
resp["Access-Control-Allow-Origin"] = '*'
resp["Access-Control-Allow-Methods"] = 'GET,PUT, OPTIONS'
resp["Access-Control-Max-Age"] = '1000'
resp["Access-Control-Allow-Headers"] = 'X-Requested-With, Content-Type'
return resp
@action(detail=False, methods=['post'])
def file_import(self, request):
try:
file = request.FILES['file'].file
path = default_storage.save(f'points/points_{datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}.csv', ContentFile(file.read()))
import_points(os.path.join(settings.MEDIA_ROOT, path))
cache.delete('points')
messages.add_message(request, messages.INFO, 'Данные успешно импортированы')
except Exception as e:
messages.add_message(request, messages.ERROR, f'Ошибка импорта: {e}')
return HttpResponseRedirect('/admin/')
@action(detail=False, methods=['get'])
def naming(self, request):
return Response(get_model_naming(models.Point), status=status.HTTP_200_OK)

@ -0,0 +1,87 @@
{% extends "admin/base_site.html" %}
{% load i18n static %}
{% block extrastyle %}{{ block.super }}<link rel="stylesheet" type="text/css" href="{% static "admin/css/dashboard.css" %}">{% endblock %}
{% block coltype %}colMS{% endblock %}
{% block bodyclass %}{{ block.super }} dashboard{% endblock %}
{% block breadcrumbs %}{% endblock %}
{% block nav-sidebar %}{% endblock %}
{% block content %}
<div id="content-main">
{% include "admin/app_list.html" with app_list=app_list show_changelinks=True %}
</div>
{% endblock %}
{% block sidebar %}
<div id="content-related">
<div class="module" id="recent-actions-module">
<h2>{% translate 'Recent actions' %}</h2>
<h3>{% translate 'My actions' %}</h3>
{% load log %}
{% get_admin_log 10 as admin_log for_user user %}
{% if not admin_log %}
<p>{% translate 'None available' %}</p>
{% else %}
<ul class="actionlist">
{% for entry in admin_log %}
<li class="{% if entry.is_addition %}addlink{% endif %}{% if entry.is_change %}changelink{% endif %}{% if entry.is_deletion %}deletelink{% endif %}">
{% if entry.is_deletion or not entry.get_admin_url %}
{{ entry.object_repr }}
{% else %}
<a href="{{ entry.get_admin_url }}">{{ entry.object_repr }}</a>
{% endif %}
<br>
{% if entry.content_type %}
<span class="mini quiet">{% filter capfirst %}{{ entry.content_type.name }}{% endfilter %}</span>
{% else %}
<span class="mini quiet">{% translate 'Unknown content' %}</span>
{% endif %}
</li>
{% endfor %}
</ul>
{% endif %}
</div>
</div>
<div>
<h3>Обновить файл точек</h3>
<form method="post" action="/api/points/file_import/" enctype="multipart/form-data">
{% csrf_token %}
<input type="file" name="file" accept=".xlsx">
<input type="submit" value="Отправить">
</form>
<h3>Обновить файл полигонов</h3>
<form method="post" action="/api/polygons/file_import/" enctype="multipart/form-data">
{% csrf_token %}
<input type="file" name="file" accept=".zip">
<input type="submit" value="Отправить">
</form>
</div>
<style>
.change_city{
appearance: none;
border: 0;
border-radius: 5px;
background: #D7DA49;
color: #fff;
padding: 8px 16px;
font-size: 16px;
}
.change_city:hover{
appearance: none;
border: 0;
border-radius: 5px;
background: #666;
color: #fff;
padding: 8px 16px;
font-size: 16px;
}
</style>
{% endblock %}
Loading…
Cancel
Save