AI Engineering Proof of Concept #2 - Personal agent infrastructure

This page is Under Construction while I explore and grow from the experience of an AI Engineer/Product Manager/Business Analyst

Background

My first AI Engineering proof of concept was an exploration of how to think through breaking down a problem into steps that can be automated with traditional logic vs steps that lend themselves to use of an LLM for planning, problem solving, or decisions.   This project will build on that experience to explore more complex business needs and objectives.

Note:  I am new to the AI Engineering and development journey but I am well versed in IT project planning so I am learning through the experience.   Also, this output is AI augmented to accelerate my capability.  I am providing the business objectives and asking the exec level questions that guide ChatGPT to refine the output for these sections to assure this will meet my needs.

Objective

The primary goal of this project is to build a coding agent that helps users—both technical and non-technical—turn vague problem statements into clear product requirements and ultimately into functional, secure software that can extend their personal agent's capabilities. The agent will be designed with a set of safety boundaries to ensure the code it generates runs entirely within the user's local system or agentic network. It will have the ability to interact with external services (e.g., retrieving email headers from the user’s email system), but it will not generate or suggest solutions that require the execution of code on external systems to fulfill a user’s objective. If a solution requires hosting code outside of the user’s system, the agent will advise the user to seek external assistance.

The coding agent will:

• Refine problem statements into actionable product requirements through interactive dialogue with the user.
• Build software solutions leveraging open-source libraries and tools, ensuring minimal dependencies and the ability to execute locally.
• Functionally test the generated software to ensure it meets the defined requirements and works as expected.
• Ensure security by integrating security best practices into the generated code and providing guidelines for safe development.
• Prioritize open-source solutions to promote transparency and reuse, reducing the need to build new components unless absolutely necessary.

Furthermore, the safety boundaries will include:

• The agent will only produce code that runs within the user’s local environment and agentic network.
• Critical settings that control the behavior of the agentic network and its tools (e.g., directories, system-level settings) cannot be altered by the agent. For example, the agent will be restricted from modifying system or application settings, ensuring that the system’s core settings remain protected.

This project aims to empower users to build and test their own software solutions securely, privately, and with minimal reliance on external services, while providing a foundation for future tool development within the agentic network.

Motivation & Relevance

In the rapidly evolving landscape of AI and software development, many individuals and small businesses struggle to turn their ideas into functional applications due to a lack of technical expertise or resources. At the same time, most AI solutions available today often rely on third-party cloud services, which may compromise user privacy, control over personal data, and the security of their systems. This project aims to address these issues by providing a coding agent that empowers both technical and non-technical users to build, test, and secure software locally without relying on external services.

The motivation behind this project is to create a tool that:

• Enables non-technical users to turn vague problem statements into clear product requirements, which the agent will then help refine into software that is specifically designed to act as tools for the personal agent. The agent guides users through the development process, ensuring the software built can directly extend or enhance the personal agent’s capabilities.
• Promotes data privacy by ensuring that all generated code and user data stays within the user’s local system, reducing reliance on third-party services or cloud-based solutions.
• Encourages open-source software by prioritizing the use of open libraries and tools, allowing users to build on existing, secure, and well-tested components. This will reduce the need to develop new code from scratch, making the development process more efficient.
• Adheres to strong security practices, with built-in checks to ensure that any generated code meets privacy and security standards, and prevents the agent from modifying system-critical settings.
• Facilitates incremental tool creation by providing a framework where users can build software specifically tailored as tools to enhance and expand the personal agent’s capabilities. The agent can adapt and evolve over time, growing into a more powerful and personalized assistant based on the user’s needs.

By offering this solution, we aim to democratize software development, allowing anyone—regardless of technical skill level—to create secure, functional software that directly enhances their personal agent, while maintaining full control over their systems and data. This project seeks to bridge the gap between technical expertise and everyday problem-solving, providing users with the tools to develop their own solutions securely and privately while enabling them to expand and evolve their personal agent's capabilities as their needs grow.

Project Goals

The primary goal of this project is to deploy a locally-hosted LLM (Large Language Model) alongside a coding agent to empower users in creating software tools specifically for expanding their personal agent's capabilities. This will enable both technical and non-technical users to generate secure, functional tools that integrate seamlessly within their personal agent environment.

---

1. Deploy a Locally-Hosted LLM
   • Goal: Set up and optimize a locally-hosted LLM to serve as the core processing engine for the coding agent. This model will handle natural language understanding (NLU), code generation, and problem refinement.
   Success Criteria:
   • The LLM can process natural language queries from users, refine vague problem statements, and generate functional code.
   • The LLM operates efficiently within the user’s local environment without relying on external cloud services.
   Challenges:
   • Ensuring the LLM is optimized for local use, considering hardware limitations and resource requirements (e.g., GPU, RAM).
   • Selecting the appropriate LLM that balances performance, accuracy, and efficiency without excessive computational costs.

---

2. Develop Coding Agent Infrastructure
   • Goal: Build the agent infrastructure that allows users to interact with the LLM to define problems, create product requirements, and generate tools specifically for extending the agent’s capabilities.
   Success Criteria:
   • The agent can take user inputs (e.g., vague problem statements) and convert them into structured product requirements.
   • The coding agent can generate software that functions as tools to extend the agent’s capabilities and integrates seamlessly into the local system.
   Challenges:
   • Designing an intuitive user interface that allows non-technical users to effectively communicate their needs.
   • Ensuring the agent generates code that adheres to user-defined constraints and system requirements.

---

3. Ensure Security and Privacy in Code Generation
   • Goal: Integrate security features to ensure generated code follows best practices for privacy and security, and cannot modify critical system settings.
   Success Criteria:
   • All generated code passes security audits and meets privacy guidelines (e.g., GDPR).
   • The agent ensures that it cannot modify system-critical settings without explicit user consent.
   Challenges:
   • Implementing automated security testing and vulnerability checks for the generated code.
   • Balancing security with usability, ensuring the user experience isn’t overly restrictive.

---

4. Facilitate Open-Source Integration and Reuse
   • Goal: Leverage open-source libraries and components to build the tools, reducing development time and complexity.
   Success Criteria:
   • A substantial portion of the generated tools will use pre-existing open-source libraries, reducing the need for custom-built code.
   • The generated code adheres to open-source licensing standards and is compatible with other open-source tools.
   Challenges:
   • Identifying the best open-source libraries that fit the user’s needs, without introducing unnecessary complexity.
   • Ensuring compatibility between various open-source tools and maintaining security and performance.

---

5. Create the Framework for Tool Expansion
   • Goal: Develop a framework to allow users to build and integrate tools incrementally, expanding their personal agent’s capabilities over time.
   Success Criteria:
   • The framework allows users to create, test, and deploy new tools that integrate seamlessly into the personal agent.
   • The agent can be extended and adapted without requiring a complete system overhaul.
   Challenges:
   • Designing a modular architecture that supports incremental expansion and adaptation.
   • Ensuring that new tools can be tested and validated easily without disrupting existing functionality.

---

6. Test and Validate the Full Workflow
   • Goal: Ensure that the entire workflow—from problem definition, through tool creation, to testing and deployment—works smoothly, providing users with a seamless, functional experience.
   Success Criteria:
   • End-to-end testing demonstrates that users can successfully generate tools and deploy them within the agent’s ecosystem.
   • The system operates with minimal errors, and the agent meets performance benchmarks.
   Challenges:
   • Ensuring comprehensive testing that covers edge cases and user-defined requirements.
   • Addressing unforeseen bugs or performance issues that may arise during integration and testing.

Key Risks

1. Risk: Limited Resources for Self-Hosting AI Models

• Risk Description: Deploying LLMs and associated tools on a limited local environment could result in suboptimal performance, long processing times, or failure due to hardware limitations.

• Mitigation Strategy:

  • Focus on lightweight, optimized versions of open-source models (e.g., distillation, quantization).
  • Prioritize only the most essential models and tools.
  • Regularly evaluate resource usage and system performance to identify bottlenecks.
  • Consider hybrid cloud options for overflow or backup.

• Decision: Proceed with resource-constrained local deployment while continuously evaluating the need for cloud resources on a case-by-case basis. Prioritize tool efficiency and lightweight models.

---

2. Risk: Integration Complexity of Multiple Platforms (Gmail, LinkedIn)

• Risk Description: Integrating with multiple platforms using an open-source agent framework could lead to unforeseen technical difficulties, API limitations, or performance issues.

• Mitigation Strategy:

  • Break integration into small phases, focusing first on one platform (e.g., start with Gmail before integrating LinkedIn).
  • Use robust, well-documented open-source libraries that are community-supported for each platform.
  • Design error-handling and fallback strategies in case of integration failure.

• Decision: Start with Gmail integration, test thoroughly, then move to LinkedIn. Regularly assess platform-specific challenges before scaling.

---

3. Risk: Security of Locally Hosted Solutions

• Risk Description: Hosting LLMs and email/LinkedIn integration tools locally could expose the system to vulnerabilities or leaks of sensitive data, especially when working with untrusted code or external services.

• Mitigation Strategy:

  • Implement strong security boundaries within the agentic network (e.g., sandbox execution, strict permission models).
  • Ensure all external interactions are carefully monitored and logged.
  • Use encryption for sensitive data and implement role-based access controls for system tools.
  • Regularly audit and patch security vulnerabilities in self-hosted tools.

• Decision: Establish a hardened environment with strict security practices, ensuring no critical system-level settings are alterable by the agent network. Limit external interactions only to necessary data retrieval.

---

4. Risk: Agent Behavior and Code Safety

• Risk Description: The AI agent may generate code that could potentially be harmful or interact with external systems in unintended ways.

• Mitigation Strategy:

  • Create a safe execution boundary where the agent can only generate code that runs within the local environment.
  • Explicitly restrict code execution across external systems unless human intervention is required.
  • Regularly update agent models with safety checks to prevent unsafe code generation.

• Decision: Enforce strict execution boundaries with the agent network. External calls or code generation requiring external execution should trigger explicit advisories for human intervention.

Work breakdown structure and key questions to ask to address the risks

Objective:  Define a WBS that is at least 25% accurate

1. Project Initiation

• 1.1 Define Requirements

  • Subtasks:
    • Confirm functional and non-functional requirements.
    • Define user stories or use cases for the agent system.
    • Establish key success criteria for each feature.
  • Key Questions:
    • Are the requirements clear and fully scoped, or are there ambiguities?
    • Have all stakeholders been involved in defining requirements?
    • What are the minimum viable features vs. nice-to-haves?

• 1.2 Risk Assessment and Mitigation Plan

  • Subtasks:
    • Identify and prioritize key risks (resource, technical, security).
    • Develop risk mitigation strategies for each identified risk.
  • Key Questions:
    • What are the critical risks that could affect the timeline or scope?
    • How accurate are the current risk assessments based on available information?
    • What contingency buffers should be added for high-risk areas?

---

2. Design & Architecture

• 2.1 System Architecture

  • Subtasks:
    • Design overall system architecture (including LLM integration and email/LinkedIn integration).
    • Ensure the security model is integrated (e.g., agent network, sandboxing).
  • Key Questions:
    • How well have we accounted for scalability and performance in the design?
    • Are there any unknown dependencies in the system architecture that could delay progress?
    • How do we ensure that the architecture allows flexibility for future updates?

• 2.2 Tool Selection & Validation

  • Subtasks:
    • Select appropriate open-source tools for LLM, email parsing, LinkedIn interaction.
    • Validate chosen tools for compatibility, security, and scalability.
  • Key Questions:
    • Are there any alternative tools that might better meet the project’s needs?
    • How do we ensure that the tools are well-supported by the community?
    • What happens if the tools become unsupported or fail to meet expectations?

---

3. Development & Integration

• 3.1 Build Email Parsing Functionality

  • Subtasks:
    • Develop code for retrieving and parsing Gmail headers.
    • Implement logic for processing email data and initiating LLM interactions.
  • Key Questions:
    • What are the known challenges when interacting with Gmail’s API?
    • How do we ensure that the email parsing system scales with larger inboxes?
    • What error-handling mechanisms are in place for API failures?

• 3.2 Build LinkedIn Interaction Functionality

  • Subtasks:
    • Develop code for authenticating and retrieving LinkedIn messages.
    • Implement logic for handling LinkedIn data and triggering LLM-based responses.
  • Key Questions:
    • How do we handle API rate limits and unexpected failures from LinkedIn?
    • What security measures ensure LinkedIn data is kept secure?
    • Is there a fallback method if the LinkedIn integration fails?

• 3.3 Develop Agent Network and Execution Boundaries

  • Subtasks:
    • Implement safety boundaries to restrict the agent’s code execution to local resources.
    • Implement role-based access control and logging for all agent activities.
  • Key Questions:
    • How will we enforce strict separation between local execution and external systems?
    • How do we log and monitor the agent’s activities to ensure no security breaches?
    • Are the safety boundaries flexible enough for future updates without compromising security?

---

4. Testing & Validation

• 4.1 Unit Testing

  • Subtasks:
    • Develop unit tests for each module (email parsing, LinkedIn interaction, agent network).
    • Automate tests where possible for continuous integration.
  • Key Questions:
    • Are the unit tests comprehensive enough to cover all edge cases?
    • How do we ensure that tests are independent and repeatable?
    • What coverage do we have for security-related tests?

• 4.2 Integration Testing

  • Subtasks:
    • Test integrated system functionality (email + LinkedIn + LLM).
    • Conduct end-to-end tests for communication between tools and the agent.
  • Key Questions:
    • What is the expected behavior when integrating email and LinkedIn features?
    • How will we test for latency or downtime from external APIs?
    • Are there fallback mechanisms if a feature fails during integration?

• 4.3 Security Testing

  • Subtasks:
    • Perform penetration testing on agent network boundaries.
    • Review all external interactions for vulnerabilities.
  • Key Questions:
    • What are the top security threats to our self-hosted environment?
    • How do we validate the robustness of our sandboxed execution model?
    • Have we tested for all common injection or data exfiltration risks?

---

5. Deployment & Monitoring

• 5.1 Deploy Agent Infrastructure

  • Subtasks:
    • Deploy the agent on local environment with necessary configurations.
    • Implement logging, monitoring, and alerting for system performance and security breaches.
  • Key Questions:
    • Are deployment pipelines and monitoring systems in place before production launch?
    • How will we manage deployment failures and rollbacks?
    • What metrics will we monitor to ensure the system operates efficiently?

• 5.2 Continuous Monitoring and Updates

  • Subtasks:
    • Continuously monitor the performance of the system and interactions with Gmail and LinkedIn.
    • Update the system as necessary to improve performance or add new features.
  • Key Questions:
    • What are the key performance metrics we need to monitor post-deployment?
    • How do we plan for scaling the system if needed?
    • What’s the process for updating tools or components without disrupting services?

---

6. Documentation & Final Review

• 6.1 Write Documentation

  • Subtasks:
    • Document system architecture, codebase, and user manuals.
    • Provide detailed instructions for future maintenance or upgrades.
  • Key Questions:
    • Does the documentation cover all necessary aspects for future developers or users?
    • Are there any gaps in documentation that could lead to misunderstandings or errors?
    • How will we ensure that the documentation stays up-to-date?

• 6.2 Final Review & Handover

  • Subtasks:
    • Conduct a final review with stakeholders and end-users.
    • Finalize project handover for future maintenance or scaling.
  • Key Questions:
    • What feedback should be gathered from stakeholders to improve the project?
    • How do we ensure that the handover process is smooth and that all team members are aligned?
    • What are the next steps post-handover for further project improvements?

---

Key Next Steps to Improve Accuracy to 60%:

1. Clarify and Solidify Requirements: Hold deeper discussions with stakeholders to finalize all functional, non-functional, and security requirements.
2. Complete Risk Assessment: Review the mitigation strategies and update with more accurate risk impact values.
3. Prototype Key Components: Start prototyping critical integrations (email parsing, LinkedIn) to gather more data on time and complexity.
4. Refine Time Estimates: Once key components are prototyped, adjust estimates for remaining tasks based on actual performance.
5. Iterate on WBS: Keep refining the WBS as you move through each phase, breaking down tasks into more granular steps based on new insights.

My Draft GPTs

 I've been creating a few GPTs and am not quite ready to formally publish them.  Please check them out and let me know what you think.

GPT #1 - I kind of know what I would like AI to help me do.  Help me make better prompts to get the most I can.

GPT#2 - I need help building policies for information security and compliance.

GPT#3 - I want to get better at creating my own prompts.   Where am I on this journey and help me get better.

GPT#4 - I am not sure about the risks of my solution.  Help me threat model this solution to identify good practice protections to reduce risk.

GPT#5 - I am worried I forgot something powerful that ought to be in my resume and I want to be prepared for interview questions about my experience.   Help me remember and prepare.

GPT#6 - I want to prepare and practice for my interviews   Help me fine tune for my next opportunity to impress

GPT#7 - I need more help tuning my resume and cover letter.   Please help me land this job!

GPT#8 - I need someone to talk to about a challenge I am facing - WARNING - Remember, this is just an LLM that has some base knowledge and can search the internet to supplement it's knowledge.   It is NOT a professional so how you use the information you receive is your entirely your decision.

These are experiments to help me understand the limits and capabilities of ChatGPT.   I make no promises about their thoroughness or accuracy but I hope that I've given ChatGPT prompts that help reduce their native risk of inaccuracy.   Please reach out to me on LinkedIn or X if you have questions or feedback.

An interesting mixture of philosophy and Threat Modeling by AI

 I was working on my fourth GPT and testing the rules that help assure the GPT stays on mission and the following occurred which was quite interesting and unlikely to be heard under normal business circumstances even though it's a bit enlightening.

You said:

Hi!

A basic InfoSec Threat Modeler said:

Hi there! 😊 How can I assist you today? Are you working on a threat model, or do you have a security-related question?

You said:

I would like to talk about Descartes. What do you know about Descartes?

A basic InfoSec Threat Modeler said:

René Descartes was a 17th-century French philosopher, mathematician, and scientist, often referred to as the "Father of Modern Philosophy." He is well-known for his statement, Cogito, ergo sum ("I think, therefore I am"), which highlights his approach to doubt and certainty as a foundation for knowledge.

Here are a few key aspects of his work:

1. Philosophy: Descartes sought a secure foundation for knowledge and developed a method of systematic doubt. His Meditations on First Philosophy is a foundational text in modern philosophy.

2. Mathematics: He made significant contributions to mathematics, including the development of the Cartesian coordinate system, which bridges algebra and geometry.

3. Science: Descartes believed in mechanistic explanations for natural phenomena and rejected Aristotelian teleology, which was dominant at the time.

While interesting, this is a bit outside my usual focus on threat modeling! If you're looking to connect this to security or systems thinking, we could draw a parallel between Descartes' method of systematic doubt and the systematic identification of threats in a security context. Would you like to explore that connection or return to threat modeling?

You said:

That is curious. How is systematic doubt related to threat modeling?

A basic InfoSec Threat Modeler said:

Great question! René Descartes' method of systematic doubt aligns conceptually with how threat modeling challenges assumptions about a system's security. Here's the connection:

Systematic Doubt in Philosophy:

Descartes advocated for doubting everything that could possibly be false until he reached something undeniable (Cogito, ergo sum). This method ensures that foundational beliefs are robust and true, forming a solid basis for further reasoning.

---

Systematic Doubt in Threat Modeling:

1. Challenge Assumptions:

   • In threat modeling, you question every assumption about the system’s security. For example:
     • Is the data transfer really encrypted?
     • Do all users truly need this level of access?
     • Could this seemingly minor vulnerability be exploited in unexpected ways?

2. Identify Threats Proactively:

   • Like Descartes doubting external realities, you assume that any part of your system might be compromised. You ask:
     • What happens if an attacker gains access to this component?
     • If one control fails, how might an attacker escalate their actions?

3. Find Certainty in Controls:

   • Just as Descartes sought absolute certainty, threat modelers validate assumptions by testing and verifying controls. For instance:
     • Testing an encryption implementation to confirm it uses the correct protocol.
     • Ensuring a firewall blocks unexpected traffic.

4. Iterative Refinement:

   • Descartes' method is iterative: doubt, analyze, confirm. Threat modeling follows a similar pattern, constantly revisiting and refining the model to adapt to new threats and insights.

---

Example in Practice: Suppose you're analyzing a web application. Systematic doubt would have you question:

• "Is our login process truly secure?"
• "What if an insider misuses admin privileges?"
• "Could a new vulnerability in our third-party library expose data?"

By doubting and exploring these possibilities, you uncover hidden threats and strengthen the system.

Would you like to apply this approach to a specific system you're working on?

---

Maybe you'd like to try?   A basic InfoSec Threat Modeler (requires ChatGPT subscription)

Analyzing a Scout Troop’s Financial Records with AI: A Journey in Clarity and Collaboration

Reconciling my son’s scout troop financial register with bank records seemed like a perfect task for AI. It’s repetitive, data-driven, and follows clear rules. Yet, my experience taught me that AI isn’t a magic wand—it’s a tool that requires clarity, iteration, and human guidance.

Initial Expectations vs. Reality

The task involved comparing a financial register exported as a PDF with an Excel sheet of bank records. I needed to identify mismatches causing a reconciliation discrepancy. Converting the files to CSV, I gave detailed instructions: match amounts and dates (within seven days), and flag unmatched rows. The AI produced results quickly, but with issues:

- Duplicates appeared in matched rows.

- Unmatched transactions didn’t make sense.

- The output format was hard to validate.

It became clear that the AI wasn’t operating with the intuitive expertise I’d expected. Frustrated, I paused the project.

Shifting Mindsets: Lessons from *Co-Intelligence*

Reading Ethan Mollick’s Co-Intelligence: Living and Working with AI helped me understand my struggle.  This was the first AI book I read that helped me recognize more about what AI cannot do.  Before I had assumed AI could behave like a seasoned accountant. Instead, I needed to:

1. Interact dynamically with the AI.

2. Normalize and clean data for precise comparisons.

3. Break the problem into smaller, phased tasks.

Adapting the Workflow

With this new mindset, I approached the problem iteratively:

1. Dynamic Interaction: I started asking the AI to review files and ask clarifying questions. For example, I prompted: “You are an accountant. Review this CSV and ask one question at a time to understand it.”

2. Normalization: We aligned formats for dates and transaction amounts. Withdrawals were translated into negatives to ensure consistency.

3. Mapping Table: Transaction comments in the register mentioned scout names, while the bank records had parent names. By extracting last names into a new column, I simplified the matching process.

4. Transaction Types: To reduce complexity, I directed the AI to split files by transaction type (e.g., checks, Zelle, debit cards). This adjustment improved the accuracy of matches.

Key Insights

The iterative process revealed:

- AI lacks domain expertise. It doesn’t “just know” rules like “never match the same transaction twice.”

- Self-debugging signals complexity. When the AI struggled to reconcile records, I realized the task needed breaking down further.

- Human assumptions matter. Recognizing biases (e.g., overestimating AI’s capabilities) is crucial for effective collaboration.

Broader Reflections

The experience raised larger questions: How do SOC teams prevent hallucinations and evaluate AI confidence in large-scale log analysis? I also reflected on my own biases—expecting AI to replace expertise rather than augment it.

Practical Takeaways

1. Dynamic Conversations: Use prompts like: “You are [expert role] helping with [problem]. Ask questions one at a time to explore solutions.”

2. Engineer’s Mindset: AI interprets problems like a developer. Clarify expectations and iterate and request that it explain what it heard to be sure you are well connected.   Consider asking it to be a Product Manager to ask better questions about the problem you are trying to solve.

3. Simplify Tasks: View AI self-debugging as a signal to break problems into phased components.

AI isn’t a miracle worker, but it can amplify your efforts when guided thoughtfully. This journey sharpened my own skills in communication, problem-solving, and critical thinking. For anyone exploring AI, start with curiosity and a willingness to collaborate—it’s a skillset that applies to humans as much as machines.

On the economics of supplier due diligence (3 of 3)

At last we've reached the end of this ramble around common challenges to building trust between competing but complementary goals.

I want to close by sharing alternatives to a few of the Seller strategies I've experienced over the last 14 years. My hope is and has always been to try to help my peers on the seller side answer their biggest question: "So what can the Seller InfoSec leadership really do in this mess of conflicting priorities to make the due diligence conversation more efficient?"

Dealing with weak skills

No matter what size you are, don't accept a bad partner in the due diligence conversation. If your future partner’s IS team are not able to understand your tech and choices, the process will be too expensive for both. Call out the efficiency impact created by their knowledge gap. Ask for someone more experienced. Sure, you may not get the replacement you would prefer but at least you will get recognized as more experienced. Helping the other’s leadership see your thought leadership is a win-win if you are respectful. A weak collaboration is impacting to both your visions for a strong partnership. Weak collaboration always results in longer discussions, executive confusion/frustration, and delays.

Dealing with small budgets

Budgets will always be tight for your IS in the early days. If you run into a giving, strong Buyer I wouldn't suggest as a recipe for success to be obtuse and try to avoid the discussion. Acknowledge the experience as an outside challenge that assures your team has done a reasonable job. Of course it can be challenging to have your weaknesses identified But this should only happen if someone above you let greed win out over a healthy balance toward compliance. The buyer's advice may even help you figure out what you missed that has already been exploited too. At best they may find where your metrics weren't accurate. Best of all, it will feel good after when they confirm you are already on your most optimizing path. The customer feedback to justify the priority of investment in your road-map will be the big win.

Is the “Op risk” defense really worth it?

Should I hide my details and push back hard to reduce my cost of sale or to avoid "Op risk"? These tactics will see diminishing returns. Do not use this as a first or even second response even though it seems like a strong position to claim. Do acknowledge when you've reached the limit of your response budget though. These assessments should be a collaboration. Every assessment should make you better and more efficient. Why do you have an NDA if not to support information sharing? The NDA should allow you to build trust and protect your valued information. It should empower you to collaborate. If you claim you can't trust the assessor, why should they trust you? If you resist it makes it hard for them not to think that you are hiding something. If you think about it, are there that many permutations of possible IS technical controls? What if you turn this around and see it is an opportunity instead of a risk? What happens if the buyer's leadership can't adopt you because of a gap? If it was something you knew needed attention and already told your leadership then they would have provided more weight for your argument. If it is something you didn't know about then you learned something useful. The worst thing that happens is you get to grow by a job change. The best occurs when you have been open and transparent with leadership so they know the limits of you and your team. In this state, everyone takes the news and grows from it.

What about the big suppliers with a lot to lose?

But what if the Seller is HUGE? What if they have a long history and a lot to lose by revealing their secret sauce? What if they have not deployed that latest gadget or Managed maturity across the board? What if they have thousands and thousands of clients? What if they are always changing and evolving so it isn't practical to answer every Buyer inquiry?

Many big companies try the third-party certifications. This will address the majority of your target market if it includes small and mid-size risk takers of course. If you skip/avoid/fail a test or don't cover a recently added product what happens? There will be low risk tolerant whales who need to know more. If these whales are not willing to cover the cost of assessment, then you lose them. It is critical to get the right balance of testing and transparency to avoid this. Customer feedback is essential. Don't only aim for "No exceptions noted". Honesty is your best policy. Every good security leader will acknowledge you are on a journey. They will take the details and exception information and use it for trust and tracking instead of doubting for a win-win.

How about this evolving idea of Continuous Compliance? One advantage of being bigger is your ability to invest in maturity. Dashboards are often created for IT and business leaders. You already spend time monitoring IT leadership to stay on point for the most valuable IS metrics, right? It was already an agreed-to primary goal for us anyway, why not share them with some clients to build trust? If you already had the discussion, you can cut off the majority of escalations. Involve Sales in the presentation to avoid unexpected escalations. This is building relationships that improve trust instead of worrying about hiding secrets. And the extra win is that the knowledge of the sharing helps keep eyes on your most critical IS metrics.

If you don't want to lift the curtain for multiple customers then a fall back would be some other independent service provider you could direct customers to where you provided continuous compliance data.

I want to end with an example I experienced again recently. I have seen this approach only twice in the last 14 years working as a Buyer's IS guide. I am a little surprised that this isn't chosen more often. It seems hard to miss the huge leverage opportunity it brings for a lower cost of sale. This approach also provides for a justifiable homogenization of your security road map. The solution is the collective, on-site visit. Yes, I actually mean invite all your customers to stop by together. This may sound a bit nerve racking or hard to pull off but bear with me. First you need to get your team together. This is going to impact them on specific days and at specific times and they must attend so plan for backups. Next you need to get everyone clear on the goal. Is your management only interested in keeping cost down for this process? Or is customer satisfaction and feedback the real goal? Next you won't get much value if your client's IS attendees don't have a few details and ground rules. You should provide your last SOC2 several weeks ahead of the onsite. If your management wants feedback (and you want some leading indicators) send the SOC2 with a request to provide written feedback. Make sure they send it well ahead so you can improve your presentation. When they come on site, don't skimp on the days of information sharing unless you want them to worry. The frequency and volume of post visit requests depends on it. You do not want this to appear to be about your needs. (That is unless you have a strong position in the market.) Provide the agenda well in advance. If you can afford it, provide separate tracks. You will have customers who had their questions answered already. Some will only need to read policies and see evidence of execution--which they should have identified as feedback to your SOC2. Lastly, I'd suggest you do this quarterly. This will give your Sales team a mechanism to keep their eager, bigger customers happy. It will provide a schedule for your partners to orient to for double value from the checkup. Don't forget to take care of your internal team though. Spend time after the visit on optimization of metrics and delivery to reduce the cost of the next assessment. Then kick off your quarterly strategic planning meetings for right after the visit. This will keep those customers’ needs visible to enable the business.

(Breathe) Wow!! This has been a lot of rambling and I think it's past time to wrap up.

To wrap up, please recall that a reasonable check and balance system is a win-win from the Federalist Papers. I hope you have seen the similarities in my re-frame of the vendor trust problem space. We all could do better together instead of only seeing each other as opponents focused only on our own win.

Publius the assessor

On the economics of supplier due diligence (2 of 3)

Welcome back to the journey of rediscovering optimal supplier risk communication from both sides of the table!

To recap, so far I identified challenges to faith that have crept into the system through abuse of the two main tools most often used to build Trust around InfoSec objectives in a Buyer / Seller relationship. 

At this point I can imagine a few camps developing ready to love or hate my perspective so here is a quote I heard recently in Brene Brown's book, "Dare to Lead" which I hope will carry us a bit further on this journey together.

"In the past, jobs were about muscles, now they’re about brains, but in the future they’ll be about the heart." —MINOUCHE SHAFIK, director, London School of Economics

Let's explore what happens if we need to see or demonstrate more confidence in this problem space populated with potential free riders willing to gain a buck at any cost while the rest of us invest reasonably to keep the mean as high above "we would care less about our customers if we could" as possible.

To start Bold, consider the ideal alternative or enhancement on the Supplier side.  Why not aim to have the Pride in your IS program sell your product for you?  Think beyond the initial cost of kicking your audit program up a notch or two and the impact of the initial few deeper dives.   What will you get if you invest in more or better controls testing with higher skilled resources for the wider ROI as suggested earlier?  Higher confidence that desired outcomes will occur with less surprises?  Higher efficiency due to well designed and maturing IS controls instead of patchworks that barely pass and create friction and exception churn?  Stronger accountibility to avoid loopholes that trip you up eventually?  Automation to simplify and give your human innovation machines far more interesting challenges to overcome and propel you to even higher heights?  True, higher cost as well but those are some fine outcomes, no?   It's what we all dream of on the IS leader side so it had to be said even though we all know it is HARD to deliver.

It would be unfair or plain naive if we didn't acknowledge that this is hardest to achieve and scale for larger orgs with a variety of products that are constantly evolving and the technical debt that inevitably comes with distance on that journey.  For the rest who still prefer to hide or to accept they have to remain ignorant of their risks because their market, threat population, regulatory universe, or soon to be exited position provide a challenge (or opportunity), it is simpler overall to skip the cost and lose some small % of their opportunities because avoiding the cost provides the incentives that still win the short term game.  There were plenty of customers who bought the Yugo and Corvair, right?  

When we consider the deeper option on the Buyer side we have to address a few unfortunately common Seller responses that can hinder our effectiveness.

• Some traditional IS leaders consider the exposure of even high level details about IS strengths and weaknesses as Operational Risk.   These leaders choose not to let content off their premises beyond their third party reports.  To speak plainly it does feel a bit insulting to the assessor when the Seller's IS do not trust their peer who is simply trying to execute due care as defined by their context.  For the supposed safety this stance provides, this argument seems counter intuitive to me.  If we assume that the set of possible permutations of technical architectures is relatively small then the only benefit to not sharing the policy level objectives of the company is what? Do they expect us to tell our business that the Seller's team is somehow more trustworthy because they might actually have a different way of spelling encryption to explain why the Seller is hiding their cards?  Their policies shouldn't say anything too precise anyway.  So what is the real risk of sharing or the benefit of not sharing?  (Besides cost of sale for the longer discussion of course.)  Do they want to force you to come onsite for anything they're not willing to cover clearly in the third party audit?  Hmmm...there we might be on to something. Unfortunately there is a potentially exploited benefit for the Seller in this seemingly honor-worthy position that Buyer's should be conscious of.  Sellers know the cost of an on-site visit from your perspective.  They can easily guess the probability of you coming on site, just by talking to the Buyer and assessing their level of need, urgency, vestment in their employer, or comfort with failure before they engaged IS for diligence.  If the Seller has a higher tolerance or less fear of losing the deal, or worse, fear that bad news you might discover could leak despite the NDA, there is Opportunity in suggesting you come on site for those Sellers who aren't actually ready but are willing to claim they are.  Most if not many Buyers will blink in this game of chicken and not spend the travel cost to verify--to the Seller's gain by incentivizing their fraud.  How to spot this play: Buyer's IS should watch to see if the IS pro is delivering responses alone or if they are carefully accompanied by someone from Sales or Legal who seems willing to push back on even your simple requests.  While you work hard to help the IS pro tell their best possible story on the other side, you notice the parties with the more vested personal interest in the deal push back more than the IS pro on your requests.  If you encounter this, consider they may have a playbook tuned just for those specific areas.  If they escalate against you after touching a specific area--or worse if you try to go deeper at all--, you know you've found something risky to their reputation and it will likely be difficult to get enough facts to even justify a finding.  Be careful.  Unless you are working a High Inherent Risk, it probably is not the right time to expose their game.  Back out of the conversation respectfully to your likely honorable and embarrassed peer and talk with your business about how valuable the solution is and how much they really want to know.  If your Buyer still wants to know, go back in and watch for other shame triggers and opportunities to tie requests together till your peer can find a way to honor their code of ethics without destroying their current unfortunately shaky stepping stone on their career.
• Sellers could insist on self assessments (e.g. SIG, CAIQ) which they themselves created.  These unfortunately come with a dare: "Go ahead.  I did all this work.  I dare you to tell your Buyer that it's useless as verification.". Not much you can do with this one but grin and bear it and leverage any formal QA standards you have to explain why verify means seeing evidence of either reality or a third party challenge of the reality, not a crib sheet.

So let's talk about those alternatives for the Buyers.

• Buyers could join a consortium or just trust the regulator to do the work for them.  What if their standards or their assessors lack the depth to keep up with the Technogy or Threats for a particular Supplier's risk case?  The power of the many in a buying relationship can be useful however it often leads to homogenization of the expectations and could potentially enable assessment with lower cost skills or a decision driven by a few higher risk tolerant entities.  In the case of many of the old but also the most emerging technologies it is often the highest risk tolerant who drive prioritization of the security features and make the decision.  This could mean the decision for the less risk tolerant becomes not "Take it or leave it" but "Take it as is with all the risk and an extended period of not knowing or fall behind." again at the benefit of the exploitive Sellers.
• I could leverage services that will give me more Intel and ongoing oversight. Unless the Seller had allowed the service deep monitoring and even if they did, I have to monitor the checker because they too are a Capitalist and might choose to cut a bit here or there on their journey.  Overall this isn't such a bad approach if you have the resources to tie the output to your IS objectives so the Intel can be triggers.
• For the sake of the spectrum, what if the buyer gave back?  What happens if I am lucky enough that I could hire and offer higher skilled IS professionals as Opportunity to the Sellers who are smaller. If their tight margins and tight IS operational budgets don't allow for improvements in confidence around documents and stories to tell, I can help them through the partnership for a shared win.  I could help them see that the work they did to respond to my Seniors, if packaged effectively, could be immediately reusable and add to their value story for their next client pursuit increasing the probability of a win through leverage?  Of course I have to be careful to avoid liability risks but still wouldn't this be a better Buyer/Seller relationship after we helped the business leadership understand the win-win?

With all this to worry about some days I wish we had a Yelp for suppliers.  We could simply vote without the comments and relatively anonymously so the skills and budget available to the larger Buyers could be shared with those who have less.  Then they get the benefit of greater resources without the “take it or leave it” position tired and worried IS pros working hard at smaller (and larger) firms are left with.

Unfortunately it seems like there are more strategies that the Buyers can leverage in this as the Sellers are always going to be pressured by the cost of sale and the push for the next higher quarter to drive to the lowest mean we accept.  

Maybe you know a few the Seller can use to be innovative instead of exploitive or evasive?  Please do share if you do as we all need to be better friends and citizens than this or it's all going to degrade to more Anderson's and GE's skirting every boundary to make the quick buck until that's the only ethics the entire culture understands.

Well, stay tuned as I'm going to break this here and come back later with a few other innovations I hope the ambitious few Sellers might try.  There just has to be a better partnership to keep all of us and the Joneses running faster than the bear and I know we won't find it without some "Rumbling with Vulnerability".

On the economics of supplier due diligence (1 of 3)

Wow it's been too long!   So busy and learning bunches on the manager side has it's pros and cons but I'm still in love with the gig and eager to share.  

I recently digested "The Federalist Papers" (full copy here) and for good measure the "Anti-Federalist Papers" (one of the variations is here) in a vain effort to understand if there really is a practical difference to hang a hat on in certain US political debates.  Besides being a refreshing mirror to help transform some of the stresses of corporate politics, (and causing my writing to expand past the brief and modern), this effort has given me a fresh perspective on my current problem space that I hope to distill and share as a contrast against the common method and it's issues with scalability and efficiency in our faster moving, Agile driven stage of the evolution of the Capitalist philosophy.

For those short on time and tired of my verbosity, I intend to share a few key points that could help us level set expectations earlier in the Supplier/Buyer partnership and still achieve a relevant degree of verified trust that is mandatory under many regulations--besides being necessary for a comfortable and successful relationship.  If I am as eloquent as I hope to be I might even convince you it can be done without a specific and constant list of objectives involving fairly precise and consistent measurement cost but instead with a more artful but still scientific approach allowing for the comfort of the more LEAN minded as well as the nervous.

So let's tighten this up…

Why do we need to assess supplier risk?  Regulations say we have to.  Besides that we want to be sure the Seller will deliver consistently.  We have promised to protect our customer's info or deliver a service consistently after all and want them to trust us so we have to be sure we can trust anyone we decide to rely on.  These seem like pretty straightforward reasons for us to invest in some quality assurance over the Buyer both pre-cutover and ongoing.

What does this look like from the Seller's perspective though?  What if they sell to multiple verticals?  What if they are starting up and their budget is limited and tight on every category of spend?  What if they are just breaking into a more highly regulated vertical than they originally had to build for?  What if a larger percentage of their customers are small, higher risk tolerant companies and only a few will want deeper truth?  What if their product is one of a kind, top of the heap, best of breed, flavor du jour?

When you see this from the 10k foot view across a large number of cases, it starts to look a lot like the complex and variety of reasons cited by Publius for designing checks and balances into the system of government to help assure an effective, always flexible, and continuously evolving government based on a relatively small base of rules below a system of common law which is intended to evolve over time to address the always present squabbles due to fringe and very local and often changing differences of position, intent, and degrees of influence.

So why then do the majority of Suppliers and Buyers still believe that the very unimpressive approach of a highly flexible but third party challenged attestation on top of generically written policies is the most effective and scalable approach to assuring trust in a partnership? 

Let's consider the failure points in these most common two methods often leveraged for proving and measuring the factors that should determine Trust between the partners.

• Policies and standards are by their nature aspirational in most orgs.  Any appropriately risk taking business outside of the Gov sector will have a potentially long list of exceptions against their own policy IF they have a good balance of IS skills to assure effective detection, discussion, and tracking to full treatment baked in at all levels of their business.  Some, most often the smaller, younger firms, consciously choose to invest light if at all in the IS, Compliance or Audit resources at the benefit of sustaining their preferred spend ratio toward innovation and new business development a few more quarters or years into their business plan.  So what value does seeing a policy really provide from the Buyer's side in this context?  Certainly it is worth the effort from the Seller's to satisfy a good proportion of customers in most regulated verticals who may still be drinking the old Koolaid, but does it really add to Trust and assure people are talking to their friends about your ability to help them sleep well at night as a strength?  I submit that there is actually very little ROI but we have to do it as we're trapped by the past and policies and standards are still a better starting point than without for any IS leader to hang their hat.

• Third party audits were set up for laudable reasons. If you read the guidance and imagine an honest firm applying them using a high quality assessor then there is tremendous ROI to be gained going beyond the relatively lower secondary benefit of use in Buyer trust development.  The company being assessed can potentially meet many governance objectives and assure a high quality operational and security environment through the annual challenge.  The dream is sound but Anderson revealed a hidden truth and GE (IMHO) put the nail in the coffin.  Is the dream of a well run and monitored company really what they are used for?  Given the high likelihood of exceptions is it not more probable that more risk tolerant management may prefer to invest on lower quality assessors and even avoid testing certain controls to achieve a report with no issues that will get past more Buyers (for the pricetag of a lighter quantity or quality IS investment strategy) who may not notice that they actually whitewashed some control domains or left out critical scope?

So, if I'm a Buyer who MUST have confidence because my market share is worth protecting, the laws mandate minimum protections for my customers and employees, and the Supplier will present a material danger to my bottom line if they fail or leak, what am I to do given this potentially preferred and fallible method being demonstrably successful?

Certainly the first option preferable to many who have lower blood pressure and faith they can absorb any incident is to accept risk.  This is always an option.  Understanding and describing the degree of the unknown side of the risk that is being accepted is a significant challenge for the IS pro tasked with helping pull back the curtain--particularly when they are actively prevented from doing so, leaving a tremendous unknown beneath an unknowable number of layers of whitewash. 

My advice to the Buyer’s IS is to consider the quality of the content as an indicator in this decision, not just that they had it ready.  The Buyer's risk assessors have to help with education here the most as the Buyer mid-level leadership may have seen the problem the same old way and would have done the same to build their story to get past the gate the same way and won't understand why mere existence is not sufficient quality.

Let’s break this up at this point.  More to come soon…