Skip to content

Importing data

Jump to bottom
Jon Froehlich edited this page Dec 30, 2021 · 19 revisions

Django makes it relatively easy to dump a database from one computer (dumpdata) and reload it on another (loaddata). This is useful and important for testing and bootstrapping your local development environment.

Because so much of the Makeability Lab website relies on artifacts (like PDF and PowerPoint files as well as thumbnails), it's not sufficient to just dump/load the database. We must also copy these files (stored in /code/media) from the exporting computer to the importing computer. I'll walk through everything below.

Dumping the data

1. Dumping the database

To dump the database data, you need to create an interactive bash shell into the Docker container and run the dumpdata command.

  1. Run the interactive shell in terminal: > docker exec -it makeabilitylabwebsite_website_1 bash

  2. From the Docker shell, run the dumpdata command. Make sure you have admin access.

    apache@43cd75e1d80c:/code$ python manage.py dumpdata > dumped_data.json
bash: dumped_data.json: Permission denied
apache@43cd75e1d80c:/code$ su
Password:
root@43cd75e1d80c:/code# python manage.py dumpdata > dumped_data.json

  3. You might also have to copy the dumped_data.json file out of the Docker container and into your local filesystem. So, go back into terminal (not the Docker shell) and type > docker cp makeabilitylabwebsite_website_1:/code/dumped_data.json .

  4. Now you should have a dumped_data.json file that you can copy over to your 2nd computer.

2. Copying the media directory

The second primary step is to copy the media directory from the Docker container into your local filesystem.

  1. From terminal (not the Docker shell), type > docker cp makeabilitylabwebsite_website_1:/code/media ., which should copy this media directory to your local filesystem
  2. I also zipped up this dir (because it can be huge) to transport it over to my 2nd computer for importing.

Importing the data

Now, on a second computer, we can import that dumped_data.json file into Django and copy over the media dir.

1. Copying over the media directory

We need to first copy over the media dir into the Docker container.

  1. From your second computer's terminal, type > docker cp ./media makeabilitylabwebsite_website_1:/code/., which will copy the media dir into the Docker container at /code/.
  2. Make sure the media dir and subdirs have appropriate permissions. I launched an interactive shell docker exec -it makeabilitylabwebsite_website_1 bash and then set all media to chmod -R 777 (but probably could use chmod -R 755). If you get an Operation not permitted error, you'll need to first switch to sudo user (e.g., > su command)
  3. In the interactive shell, you can also check that the contents have been copied over correctly

2. Importing the database data

Finally, we can import the dumped_data.json file:

  1. First, launch an interactive shell. From your second computer's terminal, type: > docker exec -it makeabilitylabwebsite_website_1 bash
  2. From within this shell, type $ python manage.py loaddata dumped_data.json and it should work. Update: I just tried this again and the command failed with a permissions issue (see below), so I had to rerun the command with sudo permissions:
No need to save, the thumbnail '/code/media/talks/images/Social_Fabrics_Designing_Wearable_Electronics_Textiles_for_Interaction_Introspection_and_Inquiry.jpg' already exists!
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 177, in load_label
    obj.save(using=self.using)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 222, in save
    getattr(self.object, accessor_name).set(object_list)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 997, in set
    self.add(*new_objs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 926, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 1099, in _add_items
    signals.m2m_changed.send(
  File "/usr/local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
    return [
  File "/usr/local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/code/website/models/talk.py", line 108, in update_file_name_talks
    os.rename(initial_path, new_path)
PermissionError: [Errno 13] Permission denied: '/code/media/talks/Social_Fabrics_Designing_Wearable_Electronics_Textiles_for_Interaction_Introspection_and_Inquiry.pdf' -> '/code/media/talks/Froehlich_SocialFabricsDesigningWearableElectronicsTextilesForInteractionIntrospectionAndInquiry_2016.pdf'

Here's an example:

apache@e86363656952:/code$ python manage.py loaddata dumped_data.json
Publication 'This is a test' has just been saved with PDF=/code/media/publications/Froehlich_ThisIsATest_ASSETS2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg' already exists!
Publication 'This is a test' has just been saved with PDF=/code/media/publications/Froehlich_ThisIsATest_ASSETS2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg' already exists!
Publication 'The Future of Global-Scale Spatial Data Collection and Analyses on Urban (in)Accessibility for People with Disabilities' has just been saved with PDF=/code/media/publications/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg' already exists!
Publication 'The Future of Global-Scale Spatial Data Collection and Analyses on Urban (in)Accessibility for People with Disabilities' has just been saved with PDF=/code/media/publications/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg' already exists!
Installed 189 object(s) from 1 fixture(s)

Clone this wiki locally