Upgrade Dedalo v3 to latest version

pricua

Hi,

We would like to upgrade a Dedalo installation that we have on a fairly old version (version 3) to the most recent version of Dedalo, so that we can keep all the information without having to re-enter it from scratch.

As is logical and normal, since it is such an old version there is no documentation on the subject. We think that we will have to upgrade progressively, first to version 4, then to v5 and finally to v6, but the truth is that we don't know where to start.

At first the folder structure is very different as well as the database, since in v3 mysql is used and from v4 onwards it starts using postgres, with a completely different structure and tables.

We understand that the most complex thing will be to make the jump from v3 to v4 and from there it should be "easier" to get to v6.

Could you guide us through this process?

Thanks!

Alex

Uffff

Yes, the v3 is old world and the last version of classical SQL model in Dédalo and this version si totally obsolete. v4 was the first implementation of a NoSQL model with an ontology and the model is totally different. In those times the change was done manually and it was updated and implemented in the projects that they wont. As community we stop support it long time ago ( +/-15 years ) and we are now focusing on the NoSQL model..

The good news is that v5 can handle the v4 updates and you can use it to update your data from v3 to v5 without install the v4.
The bad news is that the transform needs to be done manually.

And maybe you will need to work in different ways: some data can handle by importing csv files, some could be transformed using a scripts...

The main data as People, Interviews, Audiovisuals, etc. can be imported using the csv import tool.
For example, for People data, you can create a csv with the MySQL informant table and map into the ontology definition, something as:

name ---> rsc85
surname ---> rsc86

etc.

And you can create a csv file with v5 format and you can import it using the import tool in administration.

section_id	rsc85	rsc86
1	{"lg-nolan":"John"}	{"lg-nolan":"Silver"}

But the thesaurus data could be complicated in this way, so, I remember that we create a script to update it in v4.5, and maybe the most easy way is run it using the original tables. So, you will need to create the SQL tables in your PostgreSQL as:

jer_es
jer_ts
jer_on
desctiptors
etc. (you can find some other tables jer_xx)

and run the updates scripts to pass the v4.5 and finally join the result with the imported data as previous People example.

And the most complicated steps could be the indexations. I remember that this part as hard... maybe the most easy way could be to create a locator with the two indexation tables, and import it in the thesaurus as csv file.

I don't remember the v3 columns but I remember that this information was storage in two tables... and you need to combine it to create the indexation locator as:

section_id	hierarchy40
1	[{"type":"dd96","tag_id":"1","section_id":"2","section_tipo":"rsc167","component_tipo":"rsc36","section_top_id":"1","section_top_tipo":"oh1","from_component_tipo":"hierarchy40"}]

And import it into the correct section_tipo as on1, ts1, etc.

I hope it helps you.
And good look.

Best

pricua

Hi,

After a period of inactivity, we are going to try to resume this topic at @Albert (L'ETNO) request to see if it is possible to migrate the Dedalo of the Museu de la Paraula (currently in version 3) to the latest version of Dedalo and ensure that the great work done by the museum with more than 1000 transcribed and indexed interviews is not lost.

With the information provided we get a general idea of the process but we would need some more detail to be able to carry it out. I'll tell you the steps I think we should take, and you can correct me and clarify any doubts.

1 - Perform a clean installation of the latest version of Dedalo 5 (5.9.7)

2 - Import the main data using the CSV import tool within the Administration Tools.
I'm showing you the tables we have in the Dedalo 3 installation and indicating the tables that I think should be migrated this way, along with the corresponding section in Dedalo 5, in case you notice any missing or incorrect tables.

captaciones -> oh1 (Oral History) *Although it seems that some fields such as entrevistador, operador_camara or transcriptor, among others, correspond to rsc167 (Audiovisual), so two imports will likely be necessary for this table.
informants -> rsc197 (People under study)
projects -> dd153 (Projects)
reels -> rsc167 (Audiovisual)
responsable -> rsc194 (Other people)
users -> dd128 dd128 (Users) *I'm not sure if this can also be migrated using a CSV import, and if users will retain their existing username/password or if they need to be created manually

To perform this process, we need to map each column in the MySQL table to the corresponding field in the Dedalo section. If we ultimately proceed with the migration, we will likely consult you further on this, as even at first glance, we see that the previously mentioned fields for entrevistador, operador_camara y transcriptor in Dedalo 3 are fixed text, while in Dedalo 5 they are relationships to rsc194. We understand that we will need to import all the different values of these fields into rsc194 and map the IDs. We are sure that more peculiarities will arise during the migration process.

3 - Migrate all media files. To display all images (identifying image for interviews), videos, and PDF documents (certificate of cession), I understand that simply copying the files to the media directory is not enough, starting with the fact that the media directories of Dedalo3 and Dedalo5 do not correspond. Could you please explain how to copy the files, what the media directory should look like, and if anything needs to be done in the Dedalo rsc170 (image), rsc167 (audiovisual), and rsc191 (certificate of cession) sections to ensure everything displays correctly?

4 - Migrating the thesauri. You mention that a script can be used to do this using the original tables. I understand that this requires creating the following MySQL tables in the Dedalo 5 PostgreSQL database with the same columns and importing the data into them.

jer_ao
jer_ar
jer_be
jer_cl
jer_dz
jer_es
jer_fr
jer_it
jer_lg
jer_ma
jer_ts
descriptors

We tested one of them (jer_ts) and apparently nothing happened when running the script from the administration tools. (checking and unchecking the model box)

Do the thesauri need to be created beforehand in Dedalo 5? We also tried generating the thesaurus "Thematic Descriptors" with the TLD "ts" from the Hierarchy menu in case that was the issue, but the result was the same. Could you explain this process in a little more detail?

Then, also related to thesauri, we see that in the mysql database Dedalo3 we have the tables jerarquia and jerarquia_tipos, I understand that for the structuring and classification of them, we don't have to do anything with them?

5 - Transfer the interview transcripts. Import the interview transcripts, which I believe correspond to the tr table in MySQL (Dedalo3), into the rsc36 field of the corresponding audiovisual. I assume this can also be done using a CSV import, preserving all transcription and indexing marks from the interview.

6 - Migrating the indexing (the hard part). You mentioned that the indexing in Dedalo3 is in two tables, and that the information from these two tables would need to be combined into a single document and inserted via CSV into the corresponding thesaurus. I understand that these two tables are indexacion and indexacion_rel. The tables have the following columns:

The locator you mentioned needs to be generated for importing has the following format:

[{"type":"dd96","tag_id":"1","section_id":"2","section_tipo":"rsc167","component_tipo":"rsc36","section_top_id":"1","section_top_tipo":"oh1","from_component_tipo":"hierarchy40"}]

I understand that to generate it, the information from the tables needs to be combined as follows:

From the terminoID field of indexacion_rel, which has the type ts401, ts398, ts269..., the thesaurus where the CSV file will be imported is obtained. For example, for ts401, it would be ID 401 from the ts thesaurus.
The indexacionID field of indexacion_rel links it to the indexacion table, specifically the indexacionID field.
The reelID field of the indexacion table indicates the audiovisual it relates to. In the locator, it would be the section_id that goes next to "section_tipo.":"rsc167"
In the locator, section_top_id will be replaced by the ID of the interview (oral history) to which the audiovisual is associated.
In the locator, tag_id, I understand that it corresponds to the indexID column of the indexacion table.
There is another field in indexacion that I understand is not used, the timestamp field, which is a date.

7 - At this point, we would have all the information from Dedalo3 transferred to Dedalo5; we would only need to update to the latest version of Dedalo6. I assume it's not possible to update directly to the latest version; we would have to install version 6.0.0 first and then update from there to the latest. Is there a specific order to follow? Can we skip any versions, or do we have to go through them all?

If you see that we have missed something important or that something is incorrect, please let us know.

Thanks for the help