Sources and Tagging

USER GUIDE

Automatic Extraction and Classification

To ensure optimal accuracy with the Automatic Extraction & Classification feature in DEEP, it's recommended to use the "DEEP Generic NLP Enabled Framework." This framework is specially optimized for Natural Language Processing (NLP) within DEEP. To set this up:

On the DEEP home page, select 'Edit project' and navigate to the 'Analytical Framework' tab.
Click the 'NLP Friendly' button located below the search bar.
From here, choose to either select or clone the "DEEP Generic NLP Enabled Framework."

Auto extraction GIF on the DEEP platform.

It's important to note that while other frameworks are compatible with DEEP, the accuracy of the Automatic Extraction & Classification feature largely depends on how well they align with the current NLP classification models available on the platform, so make sure that you match your analytical framework and map it with the current NLP classification models available in the platform.

Once you're set, navigate to the tagging page and select the source you wish to TAG.

Using the Automatic Extraction and Classification in DEEP

Click on "Tag" to view the extracted text from the document. Then proceed to click on the "NLP extract and classify" button. If it's your first time using this function, a prompt will indicate, "Looks like you have not triggered an extraction yet." In this case, click on "Recommend entries." The system will perform two tasks: selecting entries from the document and classifying them based on your chosen framework.

After the process completes, DEEP will provide a list of entries with their primary and secondary tagging suggestions. For each entry, you have two options: "Add the entry" or "Discard the entry." Clicking "Add entry" will display a green confirmation dialog indicating successful addition. To discard an entry, select "discard entry," and it will be moved to the "Discarded Recommendations" tab.

Keep in mind, this is a machine learning model, and we're continuously enhancing its accuracy. If you come across entries that aren't relevant, please discard them. Your feedback helps refine the model's training and improves future outputs.

Once you've completed reviewing the recommendations, close the dialog. The entries will be pre-selected in the simplified text. To modify tags, simply click on a specific entry and adjust the tagging accordingly.

When you're finished, click on the blue arrow. To conclude the process, select "save." The entries will be saved and will appear on your tagging dashboard.

Tagging Page

There are two ways of accessing the Tagging page: Open a project from your Home page, where you have the summary of your projects, or directly click on the Tagging tab on top of your screen.

Project

When you arrive at the Tagging page, the first step is to choose the project you’ll be working on. Click on the drop-down menu at the top left of the screen under the DEEP logo and choose the project on which you want to work.
‍

Project source image on the DEEP platform.

Now that you have chosen your project, you have several options depending on what you want to do. Click on the following tab to complete the following action:

Sources: Upload new sources in your project and start or continue to tag based on those sources
Dashboard: Access the visualizations of all data in your project and all tagging. See the section Understanding and using data
Source groups: Only accessible if the project has an assessment registry enabled, source groups are a way of grouping sources together and create joint assessment.
Assessment registry: Access all documents registered as assessments and identify the assessment focus, capture the methodology and rate the quality. See the section Understanding and using data.
Assessment dashboard: Access the visualizations of assessments. For more, see the section Understanding and using data.
Export: Access the possibility to export the data you have tagged in an Excel or Word format and then start to analyse. For more, see the section Understanding and using data.

Add a source

After clicking on the Sources tab, access the page with all of your sources already uploaded. Helpful information is available on this page:

Total number of sources uploaded.
Percentage of sources already tagged, sources in progress and sources not tagged at all. Those can help you manage your time and workload.
Total number of entries created in the project.
Percentage of entries verified and percentage of entries controlled.

Add a source

DEEP supports a wide array of inputs to support the qualitative data review process:

URLs (and ingesting their contents)
PDFs
Word files
Excel files (limited support)
Graphics

To add a new source in DEEP, click on the corresponding insert on the right top of your screen. As defined below, there are several options to add a source.

Add a source from a website

This first option allows you to add a source without downloading.

Fill the URL and then follow these steps:

indicate a title;
add a publishing date (original one, the creation date will be done automatically);
decide who is the publishing organization and authoring organization;
assign the lead to a member of your team;
indicate its priority level, confidentiality status; and,
check the box if your source is an assessment.

Verify organizations in the search bar

The verification badge facilitates finding the relevant organization while adding documents to your projects in DEEP. While entering part of the organization's name or its acronym, DEEP automatically recognizes the organization that matches your search (both in English and other languages) and sorts them according to the number of existing sources. All organizations verified by the DEEP team appear with a verification badge. This will let you avoid duplicating organizations in your projects and be sure to be aligned with the rest of the DEEP users.

Add sources

Access this option by clicking on Add a source and then on Add sources from the menu. This option allows a “bulk upload” by picking documents from different sources (Browse Files, Drive, Dropbox).

To upload documents from drive and dropbox, make sure you are logged-in your Gmail account and give access to your documents.

When a document is uploaded, the title appears directly from the sources but can be modified. Characteristics of the document are then to be filled in the same way as for a source from a website.
You can upload several documents simultaneously. You can delete an uploaded document by clicking on the bin.

Add sources from connectors

Access this option by clicking on Add a source and then on Add sources from connectors from the menu. This way of adding sources directly refers to the section connectors on creating a project.

When your window opens, you can choose on the left hand side the project connectors and you choose which connector you’ve created that will be able to fit your documents.

Then you can start to choose documents you want in the sources found section. When you click on a source, its metadata appears on the right hand side of your window, showing you more info on the source and a preview. You can choose to ignore or add this source.

Save when finished and you have successfully added sources to your project.

Add a source directly from the Chrome extension

Adding a source is possible directly while navigating on the internet by using the deep-browser-extension on Chrome.

While on the website of the desired source, click on the extension. A window pops up where you can fill categories similar to the one presented in the previous section. Click save and your source is directly added to the corresponding project in DEEP.

Access a source

You have two options to see your sources and entries.

Option 1: Click on the Sources Table and access all of your sources

You can find the list of your sources already uploaded, their status and metadata (including date of creation, title and number of pages).

To access a particular source, click on Filter and use the function that appears on the right side of your screen. Several criteria are listed that you can use. After selecting a criterion, click on Apply and the sources will populate. You can also filter your sources according to the type of entry. Click on Clear to get back to the all list of sources or reset your filters.

Access source image on the DEEP platform.

Option 2: Click on the square and access directly the Entries Cards
‍
You have access to entries tagged and to the same system of filters as above to select entries you’re looking for. Click on an entry, you can view tags and directly edit the cards.

View source

When a new source is added to the Sources Table, the source and its metadata appear on the existing list.

‍To edit your source, click on the pen and change its metadata.

Tag your source with + Tag (For more, refer to the tagging section)

To display your entries, click on entries and access all entries already tagged in the source through the entries card viewer. For each entry, you can edit the tags, add comments, verify the entry, control the entry or delete it

Access other functionalities by clicking on the three dots. For example, you can move your lead to other projects (click on “Move to Other Projects” and look for the project you want to move the lead to), or delete your lead or mark it as In Progress.

Tagging

From the Sources Table, choose the source you want to tag and click on + Tag. On the top banner, you can access:

Source details, which will allow you to preview your source on the left insert, and information about your source on the main one.
Primary Tagging where you’ll find the framework associated with your projects and all the different categories that you can use to tag.
Secondary Tagging where you’ll be able to tag information related to location, demographic groups, affected groups, vulnerability etc…
Review where you’ll find your entries and can review primary Tagging, secondary Tagging, add comments and verify the entries.

Primary Tagging tab

The primary tagging page is divided into two spaces, one central where you find your framework to tag and one on the left where you find your source to be tagged.

On the left space, you have the original version of your source, a simplified text where you only have the text contained in your source and all the entries created.

To start tagging, you need to create an entry.

Text entries

‍To create an entry based on text, go to Simplified Text tab and select the text you have identified relevant to create an entry. When the text is selected, click on the + that appears and your entry appears.

When your entry appears, you have three possibilities:

Click on discard changes to delete this excerpt.
Click on approve changes to keep this excerpt.
Click on edit excerpt to modify it (when your entry has been edited, it is indicated on bottom left).

To assure your entry is created and to give your entry an “identity number,” approve changes and save.

‍Use text entries to help DEEP learn

‍The Natural Language Processing (NLP) system DEEP is based on learning from text. Favouring text entries rather than screenshots will help DEEP learn!

‍Image entries

‍To create an entry based on an image or infographic, go to the original tab. From this tab, you have different ways to create an entry:

Click on + and create an excerpt based on the original document. You can use this function if you don’t want to tag an infographic or image but prefer instead to describe an infographic or image of your original document. When you click on + the functions are similar to a text entry.

Click on the camera symbol and take a screenshot in your original document (you can enter full screen by clicking on the right symbol to expand and have a better view of the document and better screenshot). The functions to discard, approve changes or edit the excerpt are then similar to a text entry.

In this tab, you can also directly access the external source of the lead.

How do I tag the entry I have created?

When your entry is created (text or image), you can tag it (either directly in the simplified text tab or in the all entries tab) by using the framework that is in the central area of the interface.

Select the entry to tag by clicking on it. In the framework, tick the boxes that match your entry. Possibilities of tagging are organized according to the framework chosen for the project (for more, see the section on framework) and divided by categories.

When you have selected the tags for your entry, click on approve changes in the entry. You can go to the next entry or navigate through the list of entries you have identified for primary tagging. If you don’t want to save the tagging click on discard changes. You can also edit your entry or delete it.

How to add comments to my entries

In the third tab that presents the list of entries (All Entries), you have all the excerpts that you can modify or tag. On the top right of the entry you have access to the envelope button and can track comments and changes relative to this entry or you can add a comment to the entry and assign it to someone.

Save my entries and tags

When you’re satisfied with your entry and have approved changes, click on Save at the top right corner of your screen. You can save it at any time of the process, if you don’t, you might lose changes and not see some of your actions implemented.

After saving, you can still edit your excerpt and change your tags or even delete your entry (by clicking on the bin). You then have to approve changes and save again.

‍Secondary Tagging

When your primary Tagging is completed, you can click on the Secondary Tagging tab and access the list of your entries and comments related. The same functions for excerpts (discard, approve changes, edit and delete) are available on this page.

To add secondary tags, click on the entry you’re interested in, and select information you think are relevant in the central area of the interface by using the different widgets.

Most options of secondary Tagging are scrolling menus with different propositions or inserts to tick. Once done, click on approve changes to complete your secondary Tagging.

When you’re satisfied with your entry and have approved changes, you can click on “Save” at the top right corner of your screen and your entry is saved ! You can save at any time of the process, if you don’t, you might lose changes and not see some of your actions implemented. After saving, you can still edit your excerpt and change your tags or even delete your entry (by clicking on the bin). Approve changes and save again.

When your tagging is over on a lead, you can click on Finalize (top right of your screen) and mark the source as tagged. The status of the source will change but you still can access the Tagging and add entries. The “Finalized” status is an indicator for your overall project and team advance.

Review/Verify

After your tagging is completed you can review entries and tags of a source/lead. The verification process can happen in different places.

Verify entries through the review tab

When the primary and secondary Tagging are completed, the next tab is the review tab where you can find all the entries you’ve tagged, displayed as a type of cards where you can see and verify the excerpt, the primary and secondary tag.

You’ve verified and agree with the entry: Change the status of the entry by clicking on Verify. Tracking of the changes in entry’s status are available if you click on the envelope and access comments, you can also leave a comment to the person that has tagged or any relevant. If you want to change the status of an entry that has already been verified, you can unverify the entry by clicking on it and you must send a notification/comment to the person that has verified it the first time or any person involved in the project that seems relevant, to justify.

You want to change the primary and/or secondary tags:

For the primary tags, you can directly edit in a pop-up that only opens the concerned category.
For the secondary tags, you can directly change your tags by accessing the scrolling menus.
In both sections, you can add tags by clicking on the “+” symbol that drives you directly back to the all framework in the related section (primary or secondary).

You also have the possibility while reviewing, to apply one tag (one widget’s value) to all the entries below the one you’ve selected or to all entries created. For that, you can use the two commands that appear to the right of the category names.
‍

Verify entries directly from the Sources Table

In the list of sources, if you go to the top right of the lead you have several functionalities, one shows all the entries tagged in the source. For each entry in view, you can:

Directly review and edit the tags in the entry view (Click on “Edit tags”, access primary and secondary Tagging, change already existing tags through scrolling menu or edit in popup, or click on + and access the Tagging directly for new tags). Save your changes once done.

Add comments to an entry by clicking on comments. You open a window where actions related to the entry have been tracked and where you can add a comment and then assign it to another user of your project.

Click on “Verify” and change the entry status (either from unverify to verify or from verify to unverify with the possibility to mention the reason to unverify and flag someone to notice). The number next to the status indicates the number of times the status entry has been changed. You can track those changes in the comment section (left to the status).

Verify entries from the Entries Cards

When clicking on the Entries Cards, you access all entries related to your project in a card display. For each entry, you can click on View Tags and open an adjacent window where primary and secondary Tagging related to the entry appears. From this window, you have access to the excerpt and to all the tags related to your entry. You can review your entry and:

Change the tags by clicking on Edit Tags and access primary and secondary Tagging. You can edit your tags through (1) the popup window (next to the category), (2) changes directly in scrolling menus or other types of display available (3) by clicking on the + and directly accessing the corresponding Tagging page.

Check entry comments and track all changes and comments related to the entry by clicking on the envelope.

Click on Verify and change the entry status (either from unverify to verify or from verify to unverify. If choosing to unverify, explain the change.)

Advanced: Quality control

The Recent activity category compiles the latest activities for your projects. This includes any new entry on a source or any new comment [HP1] added by taggers on entries of your projects. You will find the name of the project and the name of the person tagging an entry or sharing a comment on the project.

By clicking on the entry or entry comment, you will be able to access the specific entry or the specific comment. In case it is a comment, you will then be able to answer the comment by clicking on the speech bubble icon.

Advanced: Assisted Tagging

To use the assisted tagging features, Assisted Tagging needs to be activated from your framework and your framework matched with the NLP.

If your framework has been already matched with the NLP, when opening the Tagging process, you’ll have an Assisted Tagging feature that appears right below the Simplified Text tab. Activate the Assisted Tagging.

Then, select an excerpt that will be your entry, instead of clicking on the + symbol, click on the purple brain icon. An entry appears with suggestions of tags (primary Tagging and secondary Tagging section).

If the tags found by the Assisted Tagging don’t seem relevant to you, you can add or unselect one or several widgets by opening the pop window next to the categories found and then create the entry by clicking on the pen, or you can keep nothing and click on the cross and discard entries.

If the tags seem relevant to you, you can click on the pen symbol and create the entry. Then, the usual process with entries apply.

This step of creating or discarding the entry will help the NLP to identify what has been relevant suggestions and what hasn’t been relevant.

If the widgets seem relevant but you want to add more tags, create the entry with the pen symbol and then tag your entry as usual.