Testing SharePoint Agent with Three Permission Levels: Evaluating Accessible Documents, Sources, and Billing

About this articleThis article was created using an automated generation workflow powered by generative AI. Based on official Microsoft documentation, we have compiled an acceptance procedure to evaluate SharePoint agents using three users with different permissions.

This article provides an acceptance procedure for running a small-scale trial of SharePoint agents. All characters, site names, document names, and numerical values are fictitious. Official Microsoft documentation was reviewed on October 28, 2026, though no operations were performed in the author's tenant. Because screen names and availability vary depending on organizational subscription, deployment status, and administrative policies, verify the behavior in your own tenant before making production changes.

Verification Status: 📘 Based on official Microsoft documentation. Tenant operations, billing, and WordPress rendering are unverified.

What to Verify First in Three Operations

1. Three users with different permissions open the same agent on a verification site prepared by the administrator.

2. All three users submit the same question and record the answers and source links.

3. Grant read permission for a single document to User B only, resubmit the same question, and compare the changes.

What to check after the operation:Observe whether the discount condition for User B changes from "Cannot be confirmed" to an answered response with sources. This is an expected result and has not been verified in an actual tenant. Do not operate in a live environment until the administrator verifies the permissions, contract, and billing conditions.

Locking Down Usage Conditions First

The reason for filling out this table first is to prevent attributing all instances of "no response" solely to permission issues. Failures such as having no entry point, being ineligible for a license or pay-as-you-go billing, .agentbeing unable to open a file, or being unable to read the data source look similar. Recording the contract, agent, and data layers separately helps avoid unnecessary permission additions.

What the Experiment Verifies

According to Microsoft documentation, SharePoint agents generate responses based on individual user data access permissions. In other words, even if three users ask the same agent the same question, their responses and supporting evidence should differ if they have access to different documents. Therefore, we compare the results of three users assuming the following four documents are placed in a fictional "Sales Proposal Pilot" site.

User A is a sales manager, User B is a regular sales representative, and User C is a test account simulating an external collaborator. Do not use real customer names, email addresses, quotation amounts, or personal information. Especially 04_個人情報一覧.xlsx is a fictitious file representing "documents that should not be shown to anyone," and there is no need to place real data there.

Items to be verified by the administrator in advance

1. Verify that the target site is for testing purposes and not a production site.

2. The three users .agentRecord whether the file can be opened. Users who cannot open it cannot proceed to the source-based answer comparison.

3. Export or document the sites, libraries, folders, and file permissions for the three users.

4. Check for the presence of a Copilot license. If testing unassigned users, the administrator checks the SharePoint agents pay-as-you-go billing, billing policies, and target security groups.

5. When using pay-as-you-go billing, determine the Azure subscription, billing policies, notification recipients, and usage suspension procedures. Reaching the budget limit alone does not stop usage or billing.

6. Determine the personnel responsible for auditing and checking usage, as well as the necessary administrative roles.

Under the principle of least privilege, granting site owner permissions to everyone from the start just to pass the test is counterproductive. Observing differences in results while maintaining different permissions is the purpose of this experiment.

Screen Operation: Send the same question to the same agent

1. Open the agent

Each test user signs in to Microsoft 365 and opens the target SharePoint site. Open the agent displayed on the site or the shared .agent file. If the entry point cannot be found, do not add permissions immediately; instead, check in the following order.

1. Whether the target account and tenant are correct

2. Whether the user is covered by a Microsoft Copilot license or a pay-as-you-go billing policy

3. .agentWhether the user has read permissions for the file

4. Whether the administrator has disabled the service plan

5. Whether there are differences in feature rollout status or organizational policies

2. Send the prompt for pasting

All three users send the following text without changing a single word.

営業担当者向けに、このサイトの製品概要、よくある質問、値引き承認条件を整理してください。
各項目について、根拠として使った文書名を示してください。
参照できない情報や根拠が不明な情報は推測せず、「確認できない」と書いてください。
個人名、メールアドレス、電話番号は出力しないでください。

The key is not to hardcode the document name in the prompt. Writing "Read 03_Price_Approval_Memo.docx" might alert the user to the existence of a document they cannot access. Convey only the business purpose and verify the names of the documents used as evidence in the results.

3. Record the answers and evidence

This is an expected result and not an actual measured value. In actual acceptance testing, not only the response body but also the citation and source links are opened to verify whether users can access the document themselves. Even if no confidential information is written in the response, if an unauthorized document can be opened from a source link, it constitutes an authorization security incident. Testing must be stopped immediately to investigate shared links, groups, inheritance, and individual permissions.

When comparing, do not base pass/fail criteria on differences in response length or phrasing. The evaluation focuses on three points: whether items required for business operations are answered, whether the basis for those answers is limited to permitted documents, and whether unknown items are avoided through speculation. Because generated text can vary even under the same conditions, requiring matching expressions blurs the original permission verification. Conversely, even if the phrasing is natural, including unauthorized pricing conditions results in a failure. Evaluators record readability and access control in separate fields to ensure security anomalies are not offset by text quality.

Retest with only one condition changed

After the initial recording, grant read permission for 03_価格承認メモ.docx only to User B. Do not change other conditions. Record the permission change request number, executor, time, and target file, wait for the changes to take effect, and then resend the same prompt.

The expected change is that only User B's discount condition changes from "cannot be verified" to an answered response with supporting evidence. Product overviews, FAQs, and personal information results should remain unchanged. If results for User A or User C also change, suspect potential issues such as granting broader permissions to a site or library instead of a single file, group changes propagating, or incorrect shared link scopes.

This comparison is not a test to measure agent intelligence. It is a controlled experiment to verify whether "a single permission change manifests as a single observable difference." Even when results do not change, do not immediately assume AI malfunction; isolate issues in the order of permission propagation, account settings, reference sources, and the query text.

Isolate failures across four layers

Restricted Content Discovery is an administrative feature that temporarily controls search and Copilot discovery scopes, and it does not revoke users' existing access rights. Do not rely solely on this feature to fix permission incidents; instead, fix the actual permissions on SharePoint. Additionally, when using Application Restrictions, treat them as an additional restriction boundary rather than a feature that grants new access to sites.

Evidence to retain through billing and auditing

During the pilot, record not only the quality of the answers, but also who verified what and when.

• Test date and time, user ID, agent name,.agentFile location

• License status for each user, pay-as-you-go billing policy, and security group

• Version of the prompt sent and storage location of the response

• Rationale for files opened, files not opened, and files unexpectedly opened

• Request, approval, and implementation records for a single condition change by User B

• Person in charge and timestamp for checking Microsoft 365 audit logs or SharePoint agent access insights

• Consumption, budget notifications, and billing policies checked in the Microsoft 365 admin center or Azure Cost Management

A budget amount is not a circuit breaker. According to current Microsoft documentation, notifications are sent based on the configured amount, but the service and billing continue even if it reaches 100%. Organizations that need the ability to stop service must decide not only where to send notifications, but also who will disconnect the pay-as-you-go service and how to handle business continuity after disconnection.

Adoption Decision

Expand the target department only when all of the following conditions are met.

• Differences in responses among the three users match the preliminary permission table.

• Neither the body text nor source links of unauthorized documents are exposed.

• Only the single condition change for User B is reflected as expected.

• .agentAnd the permission administrator for the reference source is clearly defined.

• Target users and cost monitoring methods for licenses or pay-as-you-go billing are clearly defined.

• Budget notifications are not mistaken for spending freezes, and the person responsible for halting expenditure and the operational procedure are determined.

• Audit and usage status can be verified, and the contact point for anomalies is established.

If there is even a single unexplained discrepancy, hold off on increasing users. In particular, do not conclude that it is safe just because it did not appear in the response; verify the original SharePoint permissions and source links as the definitive record.

Summary

The key to safely testing SharePoint agents is not asking the same question to three people, but observing contracts,.agent, source data, and billing as separate layers. By creating a permission table with dummy data, sending the same prompt, and changing only a single file permission, you can detect unexpected sharing within a limited scope. Because pay-as-you-go budgets trigger notifications rather than halts, you must also decide in advance on the financial aspect of "who stops it."

Official Information and Primary Sources

• Manage access to agents in SharePoint

• Set up Microsoft Copilot pay-as-you-go services

• Create a budget for pay-as-you-go billing

• Microsoft 365 pay-as-you-go pricing

• Monitor agent access to SharePoint and OneDrive

• Restricted Content Discovery

• Restrict Microsoft 365 Copilot access to sites