Best Practices for multi-source data input doc management? How do you do it?

Posted by

Had a really good question come in to yesterday’s webinar. Wanted to get your input into this so I can share with this person:

“Thoughts and best-practices on information management between multiple firms (20+). I operate in the construction sector from design to build and closeout. Having so many hands involved, incoming data is scattered at the best of times. I would love to hear how others tackle this same problem.”


Theresa, I work on mega projects within Oil & Gas, on a project and we generally have 1 main Engineering Contractor, 3 / 4 Fabricators, 100+ Suppliers and many other contractors, this is the norm for me for doc & data management. However, we need more information to understand the requirements before responding.

Engie E&P

It may or may not be of interest (or help with your question) but take a look at BIM (Building Information Modelling):

The UK’s Government Construction Strategy is now available in StratML (ISO 17469-1) format at

The relationship elements of the StratML standard enable performance partners to cross-reference their common and complementary objectives.

Eventually, value-added intermediaries will help partners monitor and track each other’s performance as well as their own.
The StratML use case for partnerships and multi-organization is documented at
The relationship between document/content/records management and strategic alignment is highlighted in the short, draft best practices paper at
Owen Ambur Chair,
AIIM StratML Committee

I will second Dennis’ request for more focus in terms of what the problem is that this person is seeking help with. That would enable a much better reply.


Hello all, I am the original inquirer!

Caveat: I’m very new to document management in general, so if any of our practices seem arcane or ridiculous – I would appreciate any suggestions you may have, I am eager to improve. My question was more so a workflow query, but I can see how the initial request is far too vague.
A little about how we operate:

We currently run a very basic online document management system, the intent is a simplified system for all members of the construction project to access docs and be kept up to date with any changes. We deal with all levels of users, from architects & engineers to the trade putting up drywall on the site – everybody needs to understand the content (usually drawings, specifications & reports) and be able to find it what they need quickly.

Each project begins with a kick off meeting where we present a small report to the pre-construction team on naming standards, syntax, fonts (TTF for OCR/Hyperlinking purposes), print drivers, submission standards, etc. We enforce these standards for the life of the project.

Each company involved is given a private folder in a staging area, that we screen prior to posting to the ‘live’ project for all to consume. Any renaming that has to occur is done via custom scripts in excel / dos. OCR & hyperlinking for AutoCAD/Revit drawings has been inconsistent across multiple platforms as well, any suggestions on how to tackle this problem would be appreciated.

The question(s):
I know we’re not optimal in how we are currently operating, I’m interested to hear how others deal with this (or similar) scenario – what software(s) do you use? have you been able to automate significant parts your process or do the inconsistent variables make that difficult? What have you found to be your workflow best-practices – for both you and your users? What red flags have you learned to steer very clear of?

I’m coming at this from a very green perspective, this still may be too vague – if so, my apologies. Regardless, thank you in advance for both your time and suggestions.

Based on the expanded information you provided I will start my feedback with 2 primary aspects:

Metadata management (and related master data management)
Workflow Metadata management – You have some aspect of this in terms of your enforcement of naming conventions and such, however, to be truly effective you need to step this up to where you have a metadata catalogue that all the external, and your own internal, players select from in order to tag content. This could be things like project, phase, structure, floor, discipline (mech, electrical, etc.) and other designators, revision designation, contractor/vendor coding, geo-coding, etc. But rather than attempting to build this into a complex naming convention string (kinda required when your process starts/ends in FTP/SFTP as I suspect it does), this should be just selectable, controlled metadata elements. This also provides you the ability to control your master data around vendors, project, location, customer(s), and so on.

Workflow – this is something you should look to automate the hell out of if you can. And having controlled metadata is key to that because you can leverage that metadata as triggers, directors, and decision point operators in your workflows. And this will hold true from FEED to walk-down and turnover stages. And yes, tagging standards for walk-down and turnover are also part of that metadata! :))

Obviously the only way to realistically achieve the above is through a good technology platform. There are some really great EDM (Engineering Document Management) solutions out there with deep functionality and control, especially from OpenText, however, I’ve found that SharePoint still strikes the best overall balance between available functionality (from appropriate 3rd party add-ons), complexity of learning curve for users and admins, and costs. Additionally, there are some great extranet manager add-ons to eliminate the FTP/SFTP conundrum that, as I said, I suspect is likely your primary interaction start/end point with your partners and customers.

Obviously there is a lot more in terms of the overall technology stack and the INTEGRATION to ha/ve truly effective and well-governed workflow than the skeleton above, but perhaps gives you some ideas?


Geoff, I am still of the opinion you have to provide project details for document management professionals to help!
Are you constructing one hotel with man suppliers of equipment, packages and software? Then we can analyse and advise how we think your document management should flow

Engie E&P

Lorne: Thank you very much for your thoughtful response! I’ve got some reading to do, thank you for the direction!

Dennis: Almost all of our clients are condominium high rise developers – this ranges from 5 storeys to 80+. Currently, we have projects at various stages (pre-construction, construction, close-out), each with different project teams (Architects, engineers, trades, etc.). O

verall structure (forgive my poor art skills):

Hi Geoff! I work for Optiform in Indianapolis, Indiana. I wonder if getting on a call with you to talk more about your current processes would be helpful. I too am new to Document Management and learning a ton (it’s so exciting!!). What you are describing sounds like a lot of the people we speak with who are using folders to manage documents. Let me tell you, using the folders on a server is much better than using the file folder system in a file cabinet, so nice job having that in place!

Companies know they need to take a more “green” approach to business by achieving a paperless office and automating workflows, but don’t always know what is available to implement until they talk with us. We handle the full scope of what you may need from the scanning, to the OCR, to the workflow and finally the document management. If you want to talk, feel free to contact me directly and I can get a call set up with one of our experts and hopefully help you understand where you should consider optimizing.

Have a great day!


We are currently undertaking a proof of concept at present. We do not have an EDRMS. All of our organisational records are held on a number of separate file shares, application file stores and Office365 folders. We are evaluating a product called Concept Searching (from the US) and Sharepoint online. Concept Searching indexes all documents on the file share, Application file store (SQL databases) & O365 environment and tags the documents with appropriate classifications/records codes. We are building a SharePoint online repository and migrating all of the seperate file stores into the new repository. Concept Searching will be used to dedup (remove ROT) and migrate the files into the new repository based on the rules we have defined. We are not required to go out for RFP so unfortunately I cant help with any documentation. I just wanted to let you know that I think you should have a look at Concept Searching 🙂

All the best with your RFP

Thanks for the responses!

Metropolitan Water District of Southern California

Hi i looked at a project like this and started with a cheap tool called Treesize to analyse a small portion of the NFS so I had some clearly definable metrics on which to base the requirements. I also used Concept Search to look at auto classification as it was moved into a new environment.
There are lots of products out there, the difficulty I found was not what to use but how to encourage the business to let go
Hope this is helpful

Lesley Holmes MA

We’d love to get a copy of your RFP when its ready. Our tools are not only used to classify the 40% of your content that has text in it, but also clean up the 60% that doesn’t. I have attached a short white paper on how your content can help build your RIM program.

Another way to let you save your indexing work and search on the file shares is using Simflofy. The power of Simflofy is that it works as a migration platform. You would be saving the indexing values in Mongodb, which is open source. Later you choose which ECM or RM platform you want to go to and use the platform to migrate the content and the index values to it.

Happy to share more on this.

Big Men On Content

Records are on file shares, SharePoints, Connections, Worksite: lots of tools. Supported by the current MiFID II and other regulatory pressures, we have been able to design a new archiving solution that focusses on accessibility of loading and retrieving the important things. We have been lucky to the support of our CIO/COO to innovate, so our backend is now a big data base with elasticstack products and proper ETL tooling. For end user access we built a custom HTML5 client, an archive rest api and a soap service. So there is no excuse to not interact with the archive. Then looking at the collaboration environment, things are indeed a bit more complex. We are the coming months trying to find a proper solution to connect these (Collaboration & Records Management) two contradictory worlds. My 5 cts today:
– Collaboration needs freedom and messiness, keep it
– load the bulk history that is not classified into a big data environment and full text index it
– integrate your favourite Collaboration tool (e.g. SharePoint/File shares) technically with your archive, so record disposition is easy
– ensure user adoption of record disposition
– use multiple models to integrate if you have multiple maturity levels –
don’t go for 100%, but start NOW


Collaboration and records management don’t need to be contradictory worlds. Traditional records management tends to make managing data, information, documents, records, etc. more difficult than it needs to be with complex rules requiring human intervention. The key is a simple classification structure based on business processes with RM requirements/business rules mapped to it.
Using a collaboration tool, such as SharePoint, build the RM rules into the process. For example: if the SharePoint site is collaborating on a project you would have two rules, one for the site itself once the project is complete (Project Type – retain x years) and another for the documents/records produced (Document Type – retain x years). Include metadata for when a document/record is final such as a flag (complete, final, record), date complete and a category (document type mapped to record series). For the Site include the close/complete date and Project Type mapped to record series).
If you have an ECM/RM this metadata can be used to manage and dispose of the site and documents. If not a workflow or manual process may be required. Key is keep it simple!

MERGE Consulting Inc

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.