Tests: Adds end-to-end tests skeleton and basic smoke test scenario (#16901)

* Chore: Adds neccessary packages

* Wip: Initial dummy test in place

* Feature: Downloads Chromium if needed

* Fix: Adds global config object

* Refactor: Adds basic e2eScenario

* Build: Adds end to end tests to config

* Build: Changes end to end job

* Build: Adds browsers to image

* Build: Adds failing test

* Refactor: Adds first e2e-test scenario

* Fix: Ignores test output in gitignore

* Refactor: Adds compare screenshots ability

* Refactor: Removes unnecessary code

* Build: Removes jest-puppeteer

* Fix: Replaces test snapshots

* Refactor: Creates output dir if missing

* Refactor: Changes aria-labels to be more consistent

* Docs: Adds section about end to end tests

* Fix: Fixes snapshots

* Docs: Adds information about ENV variables

.circleci/config.yml

@@ -69,6 +69,28 @@ jobs:
         - run:
             name: cache server tests
             command: './scripts/circle-test-cache-servers.sh'
+  
+  end-to-end-test:
+      docker:
+        - image: circleci/node:8-browsers
+        - image: grafana/grafana:master
+      steps:
+          - run: dockerize -wait tcp://127.0.0.1:3000 -timeout 120s
+          - checkout
+          - restore_cache:
+              key: dependency-cache-{{ checksum "yarn.lock" }}
+          - run:
+              name: yarn install
+              command: 'yarn install --pure-lockfile --no-progress'
+              no_output_timeout: 5m
+          - save_cache:
+              key: dependency-cache-{{ checksum "yarn.lock" }}
+              paths:
+                - node_modules
+          - run:
+              name: run end-to-end tests
+              command: 'env BASE_URL=http://127.0.0.1:3000 yarn e2e-tests'
+              no_output_timeout: 5m
 
   codespell:
     docker:

.gitignore

@@ -84,3 +84,5 @@ debug.test
 /packages/**/dist
 /packages/**/compiled
 /packages/**/.rpt2_cache
+
+theOutput/
\ No newline at end of file

README.md

-[Grafana](https://grafana.com) [![Circle CI](https://circleci.com/gh/grafana/grafana.svg?style=svg)](https://circleci.com/gh/grafana/grafana) [![Go Report Card](https://goreportcard.com/badge/github.com/grafana/grafana)](https://goreportcard.com/report/github.com/grafana/grafana) [![codecov](https://codecov.io/gh/grafana/grafana/branch/master/graph/badge.svg)](https://codecov.io/gh/grafana/grafana)
-================
+# [Grafana](https://grafana.com) [![Circle CI](https://circleci.com/gh/grafana/grafana.svg?style=svg)](https://circleci.com/gh/grafana/grafana) [![Go Report Card](https://goreportcard.com/badge/github.com/grafana/grafana)](https://goreportcard.com/report/github.com/grafana/grafana) [![codecov](https://codecov.io/gh/grafana/grafana/branch/master/graph/badge.svg)](https://codecov.io/gh/grafana/grafana)
+
 [Website](https://grafana.com) |
 [Twitter](https://twitter.com/grafana) |
 [Community & Forum](https://community.grafana.com)
@@ -12,12 +12,15 @@ Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB.
 -->
 
 ## Installation
+
 Head to [docs.grafana.org](http://docs.grafana.org/installation/) for documentation or [download](https://grafana.com/get) to get the latest release.
 
 ## Documentation & Support
+
 Be sure to read the [getting started guide](http://docs.grafana.org/guides/gettingstarted/) and the other feature guides.
 
 ## Run from master
+
 If you want to build a package yourself, or contribute - here is a guide for how to do that. You can always find
 the latest master builds [here](https://grafana.com/grafana/download)
 
@@ -48,7 +51,7 @@ go run build.go build
 
 #### Frontend assets
 
-*For this you need Node.js (LTS version).*
+_For this you need Node.js (LTS version)._
 
 ```bash
 yarn install --pure-lockfile
@@ -80,7 +83,7 @@ yarn start:hot
 env GRAFANA_THEME=light yarn start:hot
 ```
 
-*Note: HMR for Angular is not supported. If you edit files in the Angular part of the app, the whole page will reload.*
+_Note: HMR for Angular is not supported. If you edit files in the Angular part of the app, the whole page will reload._
 
 Run tests and rebuild on source change:
 
@@ -128,7 +131,9 @@ In your custom.ini uncomment (remove the leading `;`) sign. And set `app_mode =
 ### Running tests
 
 #### Frontend
+
 Execute all frontend tests
+
 ```bash
 yarn test
 ```
@@ -139,6 +144,7 @@ Writing & watching frontend tests
 - Jest will run all test files that end with the name ".test.ts"
 
 #### Backend
+
 ```bash
 # Run Golang tests using sqlite3 as database (default)
 go test ./pkg/...
@@ -150,6 +156,26 @@ GRAFANA_TEST_DB=mysql go test ./pkg/...
 GRAFANA_TEST_DB=postgres go test ./pkg/...
 ```
 
+#### End-to-end
+
+Execute all end-to-end tests
+
+```bash
+yarn e2e-tests
+```
+
+Execute all end-to-end tests using using a specific url
+
+```bash
+ENV BASE_URL=http://localhost:3333 yarn e2e-tests
+```
+
+Debugging all end-to-end tests (BROWSER=1 will start the browser and SLOWMO=1 will delay each puppeteer operation by 100ms)
+
+```bash
+ENV BROWSER=1 SLOWMO=1 yarn e2e-tests
+```
+
 ### Datasource and dashboard provisioning
 
 [Here](https://github.com/grafana/grafana/tree/master/devenv) you can find helpful scripts and docker-compose setup
@@ -171,4 +197,3 @@ plugin development.
 ## License
 
 Grafana is distributed under [Apache 2.0 License](https://github.com/grafana/grafana/blob/master/LICENSE).
-

jest.config.e2e.js

+require('module-alias/register');
+
+module.exports = {
+  verbose: false,
+  transform: {
+    '^.+\\.(ts|tsx)$': 'ts-jest',
+  },
+  moduleDirectories: ['node_modules', 'public'],
+  roots: ['<rootDir>/public/e2e-test'],
+  testRegex: '(\\.|/)(test)\\.(jsx?|tsx?)$',
+  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json'],
+  setupFiles: [],
+  globals: { 'ts-jest': { isolatedModules: true } },
+  setupFilesAfterEnv: ['expect-puppeteer', '<rootDir>/public/e2e-test/install/install.ts'],
+};

package.json

@@ -25,12 +25,16 @@
     "@types/commander": "2.12.2",
     "@types/d3": "4.13.1",
     "@types/enzyme": "3.9.0",
+    "@types/expect-puppeteer": "3.3.1",
     "@types/inquirer": "0.0.43",
     "@types/jest": "24.0.11",
     "@types/jquery": "1.10.35",
     "@types/lodash": "4.14.123",
     "@types/node": "11.13.4",
     "@types/papaparse": "4.5.9",
+    "@types/pixelmatch": "4.0.0",
+    "@types/pngjs": "3.3.2",
+    "@types/puppeteer-core": "1.9.0",
     "@types/react": "16.8.16",
     "@types/react-dom": "16.8.4",
     "@types/react-grid-layout": "0.16.7",
@@ -55,6 +59,7 @@
     "es6-promise": "3.3.1",
     "es6-shim": "0.35.5",
     "execa": "1.0.0",
+    "expect-puppeteer": "4.1.1",
     "expect.js": "0.2.0",
     "expose-loader": "0.7.5",
     "file-loader": "3.0.1",
@@ -85,6 +90,7 @@
     "load-grunt-tasks": "3.5.2",
     "mini-css-extract-plugin": "0.5.0",
     "mocha": "4.1.0",
+    "module-alias": "2.2.0",
     "monaco-editor": "0.15.6",
     "ng-annotate-loader": "0.6.1",
     "ng-annotate-webpack-plugin": "0.3.0",
@@ -94,10 +100,13 @@
     "optimize-css-assets-webpack-plugin": "5.0.1",
     "ora": "3.2.0",
     "phantomjs-prebuilt": "2.1.16",
+    "pixelmatch": "4.0.2",
+    "pngjs": "3.4.0",
     "postcss-browser-reporter": "0.5.0",
     "postcss-loader": "3.0.0",
     "postcss-reporter": "6.0.1",
     "prettier": "1.16.4",
+    "puppeteer-core": "1.15.0",
     "react-hooks-testing-library": "0.3.7",
     "react-hot-loader": "4.8.0",
     "react-test-renderer": "16.8.4",
@@ -140,6 +149,7 @@
     "tslint": "tslint -c tslint.json --project tsconfig.json",
     "typecheck": "tsc --noEmit",
     "jest": "jest --notify --watch",
+    "e2e-tests": "jest --runInBand --config=jest.config.e2e.js",
     "api-tests": "jest --notify --watch --config=tests/api/jest.js",
     "storybook": "cd packages/grafana-ui && yarn storybook",
     "storybook:build": "cd packages/grafana-ui && yarn storybook:build",
@@ -242,5 +252,8 @@
       "**/@types/*",
       "**/@types/*/**"
     ]
+  },
+  "_moduleAliases": {
+    "puppeteer": "node_modules/puppeteer-core"
   }
 }

public/app/core/components/search/search_results.html

@@ -20,7 +20,7 @@
   <div class="search-section__header" ng-show="section.hideHeader"></div>
 
   <div ng-if="section.expanded">
-    <a ng-repeat="item in section.items" class="search-item search-item--indent" ng-class="{'selected': item.selected}" ng-href="{{::item.url}}" >
+    <a ng-repeat="item in section.items" class="search-item search-item--indent" ng-class="{'selected': item.selected}" ng-href="{{::item.url}}" aria-label="{{::item.title}}">
       <div ng-click="ctrl.toggleSelection(item, $event)" class="center-vh">
         <gf-form-checkbox
            ng-show="ctrl.editable"

public/app/features/dashboard/components/AddPanelWidget/AddPanelWidget.tsx

@@ -132,10 +132,15 @@ export class AddPanelWidget extends React.Component<Props, State> {
     dashboard.removePanel(this.props.panel);
   };
 
-  renderOptionLink = (icon, text, onClick) => {
+  renderOptionLink = (icon: string, text: string, onClick) => {
     return (
       <div>
-        <a href="#" onClick={onClick} className="add-panel-widget__link btn btn-inverse">
+        <a
+          href="#"
+          onClick={onClick}
+          className="add-panel-widget__link btn btn-inverse"
+          aria-label={`${text} CTA button`}
+        >
           <div className="add-panel-widget__icon">
             <i className={`gicon gicon-${icon}`} />
           </div>

public/app/features/dashboard/components/AddPanelWidget/__snapshots__/AddPanelWidget.test.tsx.snap

@@ -35,6 +35,7 @@ exports[`Render should render component 1`] = `
       >
         <div>
           <a
+            aria-label="Add Query CTA button"
             className="add-panel-widget__link btn btn-inverse"
             href="#"
             onClick={[Function]}
@@ -53,6 +54,7 @@ exports[`Render should render component 1`] = `
         </div>
         <div>
           <a
+            aria-label="Choose Visualization CTA button"
             className="add-panel-widget__link btn btn-inverse"
             href="#"
             onClick={[Function]}

public/app/features/dashboard/components/DashNav/DashNavButton.tsx

@@ -16,7 +16,11 @@ export const DashNavButton: FunctionComponent<Props> = ({ icon, tooltip, classSu
   if (onClick) {
     return (
       <Tooltip content={tooltip}>
-        <button className={`btn navbar-button navbar-button--${classSuffix}`} onClick={onClick}>
+        <button
+          className={`btn navbar-button navbar-button--${classSuffix}`}
+          onClick={onClick}
+          aria-label={`${tooltip} navbar button`}
+        >
           <i className={icon} />
         </button>
       </Tooltip>

public/app/features/dashboard/components/SaveModals/SaveDashboardAsModalCtrl.ts

@@ -17,7 +17,7 @@ const template = `
 		<div class="p-t-2">
 			<div class="gf-form">
 				<label class="gf-form-label width-8">New name</label>
-				<input type="text" class="gf-form-input" ng-model="ctrl.clone.title" give-focus="true" required>
+				<input type="text" class="gf-form-input" ng-model="ctrl.clone.title" give-focus="true" required aria-label="Save dashboard title field">
 			</div>
       <folder-picker initial-folder-id="ctrl.folderId"
                        on-change="ctrl.onFolderChange($folder)"
@@ -34,7 +34,14 @@ const template = `
 		</div>
 
 		<div class="gf-form-button-row text-center">
-			<button type="submit" class="btn btn-primary" ng-click="ctrl.save()" ng-disabled="!ctrl.isValidFolderSelection">Save</button>
+      <button
+        type="submit"
+        class="btn btn-primary"
+        ng-click="ctrl.save()"
+        ng-disabled="!ctrl.isValidFolderSelection"
+        aria-label="Save dashboard button">
+        Save
+      </button>
 			<a class="btn-text" ng-click="ctrl.dismiss();">Cancel</a>
 		</div>
 	</form>

public/app/features/dashboard/components/ShareModal/template.html

@@ -92,7 +92,7 @@
   			</div>
   		</div>
   		<div class="gf-form" ng-show="modeSharePanel">
-  			<a href="{{imageUrl}}" target="_blank"><i class="fa fa-camera"></i> Direct link rendered image</a>
+  			<a href="{{imageUrl}}" target="_blank" aria-label="Link to rendered image"><i class="fa fa-camera"></i> Direct link rendered image</a>
   		</div>
   	</div>
 </script>

public/app/features/dashboard/dashgrid/PanelHeader/PanelHeader.tsx

@@ -90,7 +90,12 @@ export class PanelHeader extends Component<Props, State> {
           error={error}
         />
         <div className={panelHeaderClass}>
-          <div className="panel-title-container" onClick={this.onMenuToggle} onMouseDown={this.onMouseDown}>
+          <div
+            className="panel-title-container"
+            onClick={this.onMenuToggle}
+            onMouseDown={this.onMouseDown}
+            aria-label="Panel Title"
+          >
             <div className="panel-title">
               <span className="icon-gf panel-alert-icon" />
               <span className="panel-title-text">

public/app/features/dashboard/dashgrid/PanelHeader/PanelHeaderMenuItem.tsx

@@ -14,7 +14,9 @@ export const PanelHeaderMenuItem: FC<Props & PanelMenuItem> = props => {
     <li className={isSubMenu ? 'dropdown-submenu' : null}>
       <a onClick={props.onClick}>
         {props.iconClassName && <i className={props.iconClassName} />}
-        <span className="dropdown-item-text">{props.text}</span>
+        <span className="dropdown-item-text" aria-label={`${props.text} panel menu item`}>
+          {props.text}
+        </span>
         {props.shortcut && <span className="dropdown-menu-item-shortcut">{props.shortcut}</span>}
       </a>
       {props.children}

public/app/features/dashboard/panel_editor/PanelEditor.tsx

@@ -145,7 +145,7 @@ function TabItem({ tab, activeTab, onClick }: TabItemParams) {
 
   return (
     <div className="panel-editor-tabs__item" onClick={() => onClick(tab)}>
-      <a className={tabClasses}>
+      <a className={tabClasses} aria-label={`${tab.text} tab button`}>
         <Tooltip content={`${tab.text}`} placement="auto">
           <i className={`gicon gicon-${tab.id}${activeTab === tab.id ? '-active' : ''}`} />
         </Tooltip>

public/app/features/datasources/NewDataSourcePage.tsx

@@ -54,6 +54,7 @@ class NewDataSourcePage extends PureComponent<Props> {
                   onClick={() => this.onDataSourceTypeClicked(plugin)}
                   className="add-data-source-grid-item"
                   key={`${plugin.id}-${index}`}
+                  aria-label={`${plugin.name} datasource plugin`}
                 >
                   <img className="add-data-source-grid-item-logo" src={plugin.info.logos.small} />
                   <span className="add-data-source-grid-item-text">{plugin.name}</span>

public/app/features/datasources/settings/ButtonRow.tsx

@@ -12,7 +12,13 @@ const ButtonRow: FC<Props> = ({ isReadOnly, onDelete, onSubmit, onTest }) => {
   return (
     <div className="gf-form-button-row">
       {!isReadOnly && (
-        <button type="submit" className="btn btn-primary" disabled={isReadOnly} onClick={event => onSubmit(event)}>
+        <button
+          type="submit"
+          className="btn btn-primary"
+          disabled={isReadOnly}
+          onClick={event => onSubmit(event)}
+          aria-label="Save and Test button"
+        >
           Save &amp; Test
         </button>
       )}

public/app/features/datasources/settings/DataSourceSettingsPage.tsx

@@ -212,7 +212,7 @@ export class DataSourceSettingsPage extends PureComponent<Props, State> {
 
                 <div className="gf-form-group">
                   {testingMessage && (
-                    <div className={`alert-${testingStatus} alert`}>
+                    <div className={`alert-${testingStatus} alert`} aria-label="Datasource settings page Alert">
                       <div className="alert-icon">
                         {testingStatus === 'error' ? (
                           <i className="fa fa-exclamation-triangle" />
@@ -221,7 +221,9 @@ export class DataSourceSettingsPage extends PureComponent<Props, State> {
                         )}
                       </div>
                       <div className="alert-body">
-                        <div className="alert-title">{testingMessage}</div>
+                        <div className="alert-title" aria-label="Datasource settings page Alert message">
+                          {testingMessage}
+                        </div>
                       </div>
                     </div>
                   )}

public/app/features/datasources/settings/__snapshots__/ButtonRow.test.tsx.snap

@@ -33,6 +33,7 @@ exports[`Render should render with buttons enabled 1`] = `
   className="gf-form-button-row"
 >
   <button
+    aria-label="Save and Test button"
     className="btn btn-primary"
     disabled={false}
     onClick={[Function]}

public/app/features/panel/panel_directive.ts

@@ -17,7 +17,7 @@ const panelTemplate = `
           <i class="fa fa-spinner fa-spin"></i>
         </span>
 
-        <panel-header class="panel-title-container" panel-ctrl="ctrl"></panel-header>
+        <panel-header class="panel-title-container" panel-ctrl="ctrl" aria-label="Panel Title"></panel-header>
       </div>
 
       <div class="panel-content">

public/app/features/panel/panel_header.ts

@@ -34,7 +34,7 @@ function renderMenuItem(item, ctrl) {
   }
 
   html += `><i class="${item.icon}"></i>`;
-  html += `<span class="dropdown-item-text">${item.text}</span>`;
+  html += `<span class="dropdown-item-text" aria-label="${item.text} panel menu item">${item.text}</span>`;
 
   if (item.shortcut) {
     html += `<span class="dropdown-menu-item-shortcut">${item.shortcut}</span>`;