How to: run backend locally

This is if you'd like to replicate the pipeline action the_od_bods repo on a local machine.

It will gather fresh data from API calls and web scrapers, process them, then write prepared listings to the local JKAN repo.

Prerequisites

To get started, you should have the following installed:

• Git
• Python 3.x
• Python Package Manager (Pip)
• Docker

Getting started

1. Have the_od_bods and jkan repos on local in the same directory
   1. `git clone if it's the first time, or git pull to update repos
   2. you must have the jkan repo in local, even if you don't intend to use it, or the pipeline action will fail.
2. Run pipeline
   1. Run natively
      1. Set up or update Python environment
         1. the_od_bods repo runs exclusively in python
         2. python packages needed are specified in the_od_bods/requirements.txt
         3. you may use any environment manager of your choice: conda, venv, poetry or other.
      2. Run run.sh
         1. In terminal, in the_od_bods execute the shell script sh run.sh
         2. Wait for confirmation message “Scraping complete”
         3. Whole action can take ~15 minutes to complete
      3. Run merge_data.py and export2.jkan.py
         1. In terminal, in the the_ods_bods execute the shell script python merge_data.py
         2. In terminal, in the_od_bods execute the shell script python export2jkan.py
   2. Run using Docker
      1. Run image build
         1. In terminal, in the_od_bods execute the shell script docker build -t open_data_scotland_pipeline .
      2. Run image in container
         1. In terminal, in the_od_bods execute the shell script docker run -t --name ods_pipeline open_data_scotland_pipeline
      3. Extract container contents
         1. To delete previous files in the JKAN directory, in terminal, from jkan, execute the shell script rm -r _datasets/*
         2. In terminal, in jkan to copy the JKAN files, execute the shell script docker cp ods_pipeline:./usr/src/app/jkan/_datasets/ ./_datasets
         3. In terminal, in the_od_bods to copy the pipeline files, execute the shell script docker cp ods_pipeline:./usr/src/app/the_od_bods/ .
   3. To see changes in frontend see How to: Run frontend locally