Home → LIMS → LIMS Troubleshooting Procedures and Processes

LIMS Troubleshooting Procedures and Processes

IIS Configuration Analysis (No Code Access)

1. Application Pool Investigation

Check app pool settings that commonly cause LIMS issues:

# View app pool configuration
%windir%\system32\inetsrv\appcmd list apppool "YourLIMSAppPool" /config

# Key settings to check:
- processModel.idleTimeout (should be 00:00:00 for LIMS)
- processModel.maxProcesses (usually 1 for LIMS)
- recycling.periodicRestart.memory (set appropriate limit)
- recycling.periodicRestart.requests (often disabled for LIMS)
- processModel.loadUserProfile (true for file access)

2. Web.config Analysis Without Source Access

Read web.config for clues:

# Find web.config files
dir C:\inetpub\wwwroot\*.config /s

# Check connection strings (redacted for security)
findstr /i "connectionstring\|database\|server=" web.config

# Look for custom errors settings
findstr /i "customerrors\|compilation\|debug=" web.config

# Check authentication mode
findstr /i "authentication\|authorization=" web.config

3. Virtual Directory and Application Mapping

Verify application structure:

# List all sites and applications
%windir%\system32\inetsrv\appcmd list app

# Check virtual directories
%windir%\system32\inetsrv\appcmd list vdir

# Verify physical paths
%windir%\system32\inetsrv\appcmd list app "Default Web Site/LIMS" /config:physicalPath

Database Connectivity Troubleshooting

1. UDL Test (Universal Data Link)

Create test connection without LIMS:

Create new file: test.udl
Double-click to open Data Link Properties
Configure same settings as LIMS
Test connection to isolate database issues

2. SQL Server Logs Analysis

Check SQL Server error logs:

-- If you have SQL access
EXEC xp_readerrorlog 0, 1, 'login failed', 'LIMS'
EXEC xp_readerrorlog 0, 1, 'timeout', 'LIMS'
EXEC xp_readerrorlog 0, 1, 'deadlock'

3. Network Connectivity Tests

Test database network connectivity:

# Test SQL Server port
telnet sql-server-name 1433

# Test name resolution
nslookup sql-server-name

# Check firewall/connectivity
ping sql-server-name

Performance Monitoring Without Code

1. Performance Counters Setup

Key counters for LIMS applications:

# Create custom counter set
typeperf -qx | findstr /i "w3wp\|sql\|memory\|processor"

# Monitor specific counters
typeperf "\Process(w3wp)\% Processor Time" "\Process(w3wp)\Working Set" -si 10 -sc 360 -f CSV -o lims_perf.csv

Critical LIMS counters:
- \Process(w3wp*)\Working Set
- \Process(w3wp*)\Handle Count
- \ASP.NET Apps v4.0.30319(*)\Requests/Sec
- \ASP.NET Apps v4.0.30319(*)\Request Wait Time
- \Web Service(*)\Current Connections

2. Failed Request Tracing (FREB)

Enable detailed tracing for errors:

Enable Failed Request Tracing in IIS Manager
Configure tracing rules:
Status codes: 500-599, 400-499
Time taken: >30 seconds
All requests for specific URLs
Analyze XML trace files in: %SystemDrive%\inetpub\logs\FailedReqLogFiles

3. ETW (Event Tracing for Windows)

Capture detailed IIS events:

# Start ETW trace
logman create trace "LIMS-IIS" -p "Microsoft-Windows-IIS-Logging" -o C:\temp\iis-trace.etl -ets

# Stop trace after issue reproduction
logman stop "LIMS-IIS" -ets

# Convert to readable format
tracerpt C:\temp\iis-trace.etl -o C:\temp\iis-trace.csv -of CSV

File System and Permissions Analysis

1. LIMS File Access Patterns

Common LIMS file locations to monitor:

# Check temp directories
dir C:\Windows\Temp\*.* /o-d | head -20
dir %USERPROFILE%\AppData\Local\Temp\*.* /o-d | head -20

# Check upload/processing directories
dir C:\inetpub\wwwroot\uploads\*.* /o-d | head -20
dir C:\inetpub\wwwroot\temp\*.* /o-d | head -20

# Look for large files that might cause timeouts
forfiles /P C:\inetpub\wwwroot /S /M *.* /C "cmd /c if @fsize gtr 104857600 echo @path @fsize"

2. Permission Analysis

Check application pool identity permissions:

# Find app pool identity
%windir%\system32\inetsrv\appcmd list apppool "YourLIMSAppPool" /text:processModel.identityType

# Test file access as app pool user
runas /user:IIS_IUSRS cmd
# Then test file operations from that context

Network and Integration Monitoring

1. LIMS Integration Points

Common integration failure points:

Instrument connections: Check TCP/UDP ports, serial connections
File watchers: Monitor input directories for data files
Web services: Test SOAP/REST endpoints
Email notifications: SMTP connectivity and authentication
Document management: File server connectivity

2. Port Monitoring

Monitor network connections:

# Check active connections
netstat -an | findstr :80
netstat -an | findstr :443
netstat -an | findstr :1433

# Monitor specific process connections
netstat -ano | findstr [w3wp-pid]

3. SSL/Certificate Issues

Common LIMS certificate problems:

# Check certificate store
certlm.msc

# View certificate details
certutil -store MY
certutil -store Root

# Test SSL connectivity
openssl s_client -connect yourserver:443 -servername yourserver

Error Pattern Recognition

1. Common LIMS Error Signatures

Look for these patterns in logs:

Sample upload failures: 413, Request Entity Too Large, file size errors
Database timeouts: timeout, deadlock, connection pool exhausted
Session state issues: ViewState, session timeout, authentication redirects
Integration failures: API timeouts, instrument communication errors
Memory leaks: Gradual memory increase, eventual crash
File locking: File access denied, sharing violations

2. Time-based Analysis

Correlate errors with usage patterns:

# Group errors by hour to find peak load issues
# Look for patterns related to:
- Daily sample processing batches
- Instrument data imports
- Report generation schedules
- Backup operations
- Maintenance windows

Proactive Monitoring Setup

1. Health Check URLs

Create monitoring endpoints:

Database connectivity test page
File system access test
Integration service status
Memory usage indicator
Application version/build info

2. Log Rotation and Retention

Prevent disk space issues:

# Setup log cleanup job
forfiles /P "C:\inetpub\logs\LogFiles" /S /M *.log /D -30 /C "cmd /c del @path"

# Monitor disk space
wmic logicaldisk get size,freespace,caption

3. Alerting Thresholds

Set up monitoring for:

Memory usage >80% for 5+ minutes
Response times >30 seconds
Error rate >5% in 10-minute window
Disk space <10% remaining
Failed database connections
App pool recycles

Emergency Response Procedures

1. LIMS Application Down

Quick recovery steps:

Check app pool status and restart if needed
Verify database connectivity
Check recent Windows updates/patches
Review last 2 hours of logs for clues
Test with minimal user account
Check file system permissions
Verify certificate validity

2. Performance Degradation

Investigation sequence:

Check current memory/CPU usage
Analyze recent slow requests
Test database query performance
Check for file system issues
Review integration point status
Monitor active user sessions

3. Data Integrity Issues

When LIMS data seems corrupted:

Stop further data processing
Check database consistency
Review file upload logs
Verify backup integrity
Check integration data sources
Document timeline of events

Tools and Utilities Reference

Built-in Windows Tools

Event Viewer: Application, System, Security logs
Performance Monitor: Real-time counters and data collector sets
Resource Monitor: Process and resource usage
IIS Manager: Configuration and real-time monitoring
Services Console: Service status and dependencies

Command Line Utilities

appcmd: IIS configuration management
logman: ETW trace management
typeperf: Performance counter data collection
netstat: Network connection monitoring
tasklist/taskkill: Process management

Third-party Tools (if available)

Process Monitor: File/registry access monitoring
Wireshark: Network traffic analysis
SQL Profiler: Database query monitoring
LogParser: Advanced log file analysis

Last updated: 2025-08-26 20:00 UTC