I finally got the chance to try out the new agent mode in ChatGPT for a test use case. I’ve been experimenting with agent mode in ChatGPT and watching related videos. On a recent experiment, I used it to search for flight tickets with some surprising findings. I think this new capability allows end users to explore the benefits of AI in new ways. It expands the scope of where AI agents can operate.
I’ve been wondering whether agent mode could be used to test a web part in SharePoint. In many product organizations I’ve worked for, UI testing is an essential part of complementing feature development. I particularly like to run UI tests myself on features that I ship. While test teams do a great job, running my own tests has sometimes helped me uncover bugs that should never have made it to production.
In my experiment, I used ChatGPT’s agent mode to test the out-of-the-box (OOTB) weather web part in SharePoint. In this blog post, I’m sharing my experience and findings.
What is agent mode?
Agent mode in ChatGPT allows AI to open a browser (on a cloud computer, not your own machine) and access webpages just like a human would. It combines deep research with the ability to take actions based on your instructions. This means AI can interact with a website even if it doesn’t have an API or connectors, simply by navigating the UI.
For example, you can use agent mode to scan job portals for specific roles and document the results in a spreadsheet, or compare competitors’ pricing with data in a file, then send you a notification when the price changes.
Does Microsoft 365 Copilot support agent mode?
Yes, you can use the Computer use feature in Copilot Studio (currently in preview) to instruct your agent to interact with any system via the desktop or browser UI. This works similarly to ChatGPT’s agent mode, but with the advantages Microsoft 365 Copilot provides.
You can’t enable Computer use directly from the Agent builder in Microsoft 365 Copilot. Instead, you must create an agent in Copilot Studio and meet the requirements for this functionality. For business-related scenarios, I would generally choose Copilot over ChatGPT.
How agent mode works for testing a SharePoint web part
In my test tenant, I created a new, unlicensed user and gave it access to a SharePoint page containing only the OOTB weather web part.
I then created the following prompt with some instructions and test scenarios. Detailing the steps to find the edit button for the web part properties saved time and helped the agent avoid trial and error.
You are acting as a quality assurance tester for a SharePoint site.
Open this SharePoint page: [URL].
Locate the "Weather" web part on the page.
Test the following:
- The weather web part shows a valid location.
- The current temperature is displayed in Celsius or Fahrenheit.
- The forecast is visible for at least the next three days.
- All icons, text, and styling load correctly (no broken images or overlapping text).
- The data refreshes or updates correctly if the page is reloaded.
In the web part properties, test the following:
- Switching the "Display temperature as" property between Fahrenheit and Celsius reflects the correct value on the web part.
- Turning the "Show title" property on and off shows or hides the web part title.
Where to find the icon for the web part properties: On the UI, after you put the entire page in edit mode, the web part edit icon is located in the toolbar just above the weather widget. It’s the second icon from the left, right between the six-dot “drag handle” icon and the clipboard icon. The icon itself shows two horizontal lines with small circles on them, resembling audio mixing sliders or filter adjustments.
Report back in a clear, structured format (including Pass/Fail for each test point) and note any discrepancies found, along with suggestions for improvements. Use PowerPoint for your report.
Be thorough and note any minor issues with layout, data accuracy, or performance.
The agent spun up a cloud machine and prompted me to sign in to the SharePoint site. After signing in, it located the web part and ran the validations I specified. In the end, the agent documented its findings in a PowerPoint deck, including the following table of test results.
| Test item | Outcome | Notes |
|---|---|---|
| Location displays valid city & country | Pass | London, Greater London, United Kingdom shown. |
| Temperature unit shown (°F/°C) | Pass | Displays 74 °F; toggling to °C shows 23 °C. |
| Forecast visible for ≥3 days | Fail | Only current day shown; multi-day forecast missing. This was expected since the web part doesn’t support weather forecast. |
| Icons & styling load correctly | Pass | Weather icon and text render cleanly. |
| Data refresh on reload | Pass | Web part loads correctly after refresh. |
| Property “Display temperature as” | Pass | Switching between °F and °C updates the temperature accordingly. |
| Property “Show title” | Pass | Turning off hides the header; turning on restores it. |
How secure is this?
Be very careful about the SharePoint account you use for this type of automation. The less access it has, the better. For this specific test, create an impersonated account, never use your own, and make sure MFA is enabled. Again, be as restrictive as possible with the permissions and site access for your dummy account. This will help prevent ChatGPT from potentially reading sensitive tenant data.
Even with MFA enabled, the access token from an agent mode session could still be used to access your SharePoint environment. To reduce risk, keep the account’s permissions minimal or consider creating a separate tenant entirely. This ensures no PII (personally identifiable information) or other sensitive content is exposed.
OpenAI itself warns that sharing credentials during an agent mode session might put your security at risk, so keep that in mind!
What about the Everyone Except External User group
Even unlicensed users in SharePoint are members of the Everyone Except External Users group, which gives them access to all SharePoint sites where this group is granted permissions. In Vlad’s SharePoint blog, author explains how to remove a user from the “Everyone Except External Users” group in SharePoint by making the user an internal guest.
To apply this setting, navigate to Microsoft Entra ID and change the User type from “Member” to “Guest”. You’ll also need to update your site settings to allow existing guests; otherwise, your dummy account won’t be able to access your test site.
Closing thoughts
I’m impressed with how easy this was to set up and with the AI’s ability to navigate the UI according to my instructions. The agent correctly validated the web part and changed properties without issues.
The more precise you are in your guidance, such as telling it exactly where to find the edit button and what UI elements to validate, the better your results will be.
I don’t see agent mode as a replacement for manual testing or the processes and tools you already use. But it’s a neat addition for final testing phases for example, and, for tech enthusiasts, a fun experiment. I’m curious about your use cases and look forward to discovering more scenarios where agent mode can be applied.
Thanks for reading,
Jarbas
