We are dealing with the painful task of integrating Generative IA in our architecture, applications, etc. To cope with regulation we need to keep an inventory of all of our IA models. What are the attributes you would (or are already using) to maintain such inventory? Could you share what attributes do you believe that are relevant to decide witch LMM is right for the problem you want to solve?
Sort By:
Oldest
Senior Data Scientist in Miscellaneous9 months ago
Hi,frankly speaking, I'd less ask for attributes. Instead I'd check, if your models are relevant to the EU and US AI acts. Maybe the NIST AI Risk Management Framework (see: https://www.nist.gov/itl/ai-risk-management-framework) might help you in a comprehensive way.
Good luck!
Lead Enterprise Architect in Energy and Utilities7 months ago
Hi!As I’ve promised, here you have the list of attributes that we have included in the inventory so far. Bear in mind that we must considered all IA models, no matter if they are generative or not. Attributes with “*” are mandatory.
Name*: // Name of the model. Give to your model a cool name.
Model objective*: // One of these:
Classification
Grouping
Forecast
Optimization
Document efficiency
Relational efficiency
Content generation
Detection
User interaction
Other: Indicate
Description*: // Text with a brief description about what is the purpose of this model.
Risk Level*: // According to the EU regulation
Inadmissible // Obviously this should never be used.
High
Limited
Minimum
Security and Ethics*: // Text. Any consideration regarding to security and ethics that must be taken into account.
Deployment*: // Where is this model deployed?
SaaS
On-prem
Cloud-AWS
Cloud-Google
Cloud-Azure
Other: Indicate
Use (Internal/External)*: // Is this model used only for internal employee or can it be accessed outside the company?
Application that uses the model: // Text. Usually, a model doesn’t run by itself, it is wrapper by an application. Name it here. Application portfolio (another inventory) should point also to this model.
Application(s) in which it is used*: // Text. The output of the model could be used by others applications. Name them here.
State*: // What is the state of this model regarding lifecycle?
Development
Test (SQA)
Production
Retired
Responsible in the company*: // Who is in charge of this model *inside* the company.
Intellectual property*: // Has the company the IP of the model? Is it shared with other company? Is it 100% other company IP?
Documentation: // Points to a document repository where this model documentation is stored.
Family: // Model family. For now we are focused in just two: “AI” / “Gen AI”.
Model: // One of these:
Regression
Decision tree
K-Means
Neural networks
Deep learning
Genetic algorithms
Natural language processing
Foundational
Adversarial networks
Transformers
Other: Indicate
Programming language used: // In what programming language is this model developed
Libraries: // List of libraries that this model needs to run. Use a plain list of a requirements.txt file.
Performance Metrics: // Indicate this model performance using a well know metric:
Mean square error
Root mean square error
Mean absolute error
Absolute median error
Determination coefficient (R2)
Precision
Recall
F1-Score
Other: Indicate
Hyperparameters: // List and values of the models’ hyperparameters.
Seed: // Indicate the seed’s value. If it is a random value, just use “random”.
Source code path: // If you have this model source code, put here a reference to the code repository (could be a link to github)
Prediction frequency: // How often this algorithm runs to create a prediction (or output)
Retraining frequency: // How often this algorithm is retrained.
Last training date: // When was the last time this algorithm retrained.
Costs: // Computational cost of running this algorithm.
Development date: // When was this algorithm developed.
Hardware / Software requirements: // Is there a special requirement to run this model (GPU, Memory, etc.)
License: // use this field if there are special consideration regarding licensing.
Input data: // For each source of data: Name of the source, location (ftp, api, etc.), format, security concerns.
Output data: // for each output: Name of the output, location, format, security concerns.
Training data: // *same as input data*
I'll update this post with any new information that we should add (or remove!) from the inventory
Lead Enterprise Architect in Energy and Utilities3 months ago
Hi!The latest version of the European regulation document gives a detailed explanation of the necessary technical documentation. All the information is in article 11 and annex IV. It must be taken into account that this need for documentation applies only to high-risk systems and that there is no distinction between types of AI (Generative or not).
Best regards,,
Senior Data Scientist in Miscellaneous3 months ago
Aware of th eimpact of the EU AI Act, there is an ongoing initiative to assist the companies in dealing with the consequences out of. See: https://digital-strategy.ec.europa.eu/en/policies/ai-pact
I have been integrating GenAI into my Enterprise Automation work by having the automation prompt the LLM and using Automation in this way to perform and create an Auditable process using AI.
I call this IA and GenAI use-case AI-Spanning. As a member, you should be able to contact me if you need more information.