Мы используем файлы cookie для быстрой и удобной работы сайта. Выберите, какие файлы cookie вы разрешаете нам использовать. Подробнее в Политике конфиденциальности.
Обсудить проект
Отправьте запрос, и наши специалисты свяжутся с вами в течение 1 часа
Нажимая кнопку «Отправить», Вы даете однозначное согласие на обработку своих персональных данных в объеме и с целями, определенными Политикой в отношении обработки персональных данных.
Development of an analytical system for recording and verifying waste removal trips for compliance with criteria
development of enterprise data warehouses and data lakes
construction
1.5
30%
the accuracy of criteria calculation has been increased
by
by
the speed of criteria processing and flight display window generation has been increased
times
Customer
A large construction company in the Moscow region
CHALLENGES/FEATURES
Checking the flight for compliance with 22 criteria: the presence of geocoordinates for the vehicle for every 15 seconds of the flight, the location of the vehicle no further than 100 m from the testing area at the end of the flight, etc.
implementation on an open-source stack based on Apache NIFI, Airflow, Greenplum, and Apache Superset
the possibility of horizontal scaling of the solution
3 years of data processing in the system with a storage capacity of 5 TB
To improve the quality, transparency of accounting, and assessment of the success of waste removal trips to landfills in the Moscow region
Task
solution
Technical solution
1. We implemented processes for loading data from heterogeneous sources to the ODS (operational raw data) layer based on the Apache NiFi data transport framework for 4 sources and 18 entities.
2. Implemented the formation of a detailed DDS layer from ODS for 18 entities based on calling MPP-DBMS functions for building corporate data warehouses Greenplum and DAGs Airflow
3. Developed a daily DAG (acyclic graph for building data processing pipelines) that calculates the correctness of flights daily by calling 51 functions in 25 minutes, checking 22 flight success criteria.
4. Created a custom DAG with the ability to calculate flights for arbitrary dates
2. Implemented the formation of a detailed DDS layer from ODS for 18 entities based on calling MPP-DBMS functions for building corporate data warehouses Greenplum and DAGs Airflow
3. Developed a daily DAG (acyclic graph for building data processing pipelines) that calculates the correctness of flights daily by calling 51 functions in 25 minutes, checking 22 flight success criteria.
4. Created a custom DAG with the ability to calculate flights for arbitrary dates
Result
Business values
Import independence of software
a system for recording the correctness of flights based on scaled independent import-substituting technologies
Flexibility
A flexible solution allows for recalculating the correctness of a flight for any arbitrary date without changing the code.
Speed
The speed of criteria processing and flight display creation has increased by 30%
Accuracy
increased transparency in the accounting of correct flights
We integrated four data sources, which are ready for reuse in other projects and the development of approaches to data management and the Data Office throughout the organization.
The accuracy of criteria calculations has increased by 1.5 times due to the use of proprietary algorithms and plugins
By clicking the "Submit" button, you expressly consent to the processing of your personal data to the extent and for the purposes defined in the Personal Data Processing Policy.
Development of software
and Big Data solutions
and Big Data solutions
Send a request and our specialists will contact you within 1 hour.
Choose a convenient method of communication
You can attach three files up to 3 MB each. Formats: doc, docx, pdf, ppt, pptx
Сообщение об успешной отправке!