This commit is contained in:
Tutur33
2023-11-24 23:58:26 +01:00
parent 25395c0ee1
commit 938ad9d309
4191 changed files with 41 additions and 518781 deletions
-15
View File
@@ -1,15 +0,0 @@
lib-cov
*.seed
*.log
*.csv
*.dat
*.out
*.pid
*.gz
pids
logs
results
node_modules
npm-debug.log
-8
View File
@@ -1,8 +0,0 @@
sudo: false
language: node_js
node_js:
- "6"
- "8"
- "10"
before_install:
- npm install -g npm@2
-23
View File
@@ -1,23 +0,0 @@
Copyright 2012 Thorsten Lorenz.
All rights reserved.
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
-206
View File
@@ -1,206 +0,0 @@
# redeyed [![build status](https://secure.travis-ci.org/thlorenz/redeyed.svg?branch=master)](http://travis-ci.org/thlorenz/redeyed)
<a href="https://www.patreon.com/bePatron?u=8663953"><img alt="become a patron" src="https://c5.patreon.com/external/logo/become_a_patron_button.png" height="35px"></a>
*Add color to your JavaScript!*
![frog](http://allaboutfrogs.org/gallery/photos/redeyes/red1.gif)
[Red Eyed Tree Frog](http://allaboutfrogs.org/info/species/redeye.html) *(Agalychnis callidryas)*
## What?
Takes JavaScript code, along with a config and returns the original code with tokens wrapped and/or replaced as configured.
## Where?
- server side using nodejs
- in the [browser](#browser-support)
## What for?
One usecase is adding metadata to your code that can then be used to apply syntax highlighting.
## How?
- copy the [config.js](https://github.com/thlorenz/redeyed/blob/master/config.js) and edit it in order to specify how
certain tokens are to be surrounded/replaced
- replace the `undefined` of each token you want to configure with one of the following
### {String} config
`'before:after'`
wraps the token inside before/after
### {Object} config
`{ _before: 'before', _after: 'after' }`
wraps token inside before/after
#### Missing before and after resolution for {String} and {Object} config
For the `{String}` and `{Object}` configurations, 'before' or 'after' may be omitted:
- `{String}`:
- `'before:'` (omitting 'after')
- `':after'` (omitting 'before')
- `{Object}`:
- `{ _before: 'before' }` (omitting '_after')
- `{ _after: 'after' }` (omitting '_before')
In these cases the missing half is resolved as follows:
- from the `parent._default` (i.e., `Keyword._default`) if found
- otherwise from the `config._default` if found
- otherwise `''` (empty string)
### {Function} config
`function (tokenString, info) { return {String}|{Object}; }`
#### Inputs
- tokenString: the content of the token that is currently being processed
- info: an object with the following structure
```js
{
// {Int}
// the index of the token being processed inside tokens
tokenIndex
// {Array}
// all tokens that are being processed including comments
// (i.e. the result of merging esprima tokens and comments)
, tokens
// {Object}
// the abstract syntax tree of the parsed code
, ast
// {String}
// the code that was parsed (same string as the one passed to redeyed(code ..)
, code
}
```
In most cases the `tokenString` is all you need. The extra info object is passed in case you need to gather more
information about the `token`'s surroundings in order to decide how to transform it.
See: [replace-log-example](https://github.com/thlorenz/redeyed/blob/master/examples/replace-log.js)
#### Output
You can return a {String} or an {Object} from a {Function} config.
- when returning a {String}, the token value will be replaced with it
- when returning an {Object}, it should be of the following form:
```js
{
// {String}
// the string that should be substituted for the value of the current and all skipped tokens
replacement
// {Object} (Token)
// the token after which processing should continue
// all tokens in between the current one and this one inclusive will be ignored
, skipPastToken
}
```
### Transforming JavaScript code
***redeyed(code, config[, opts])***
Invoke redeyed with your **config**uration, a **code** snippet and maybe **opts** as in the below example:
```javascript
var redeyed = require('redeyed')
, config = require('./path/to/config')
, code = 'var a = 3;'
, result;
// redeyed will throw an error (caused by the esprima parser) if the code has invalid javascript
try {
result = redeyed(code, config);
console.log(result.code);
} catch(err) {
console.error(err);
}
```
***opts***:
```js
{ // {Boolean}
// if true `result.ast` property contains the abstract syntax tree of the code
// if false (default) `result.ast` is not assigned and therefore `undefined`
buildAst: true|false
// {Boolean}
// if `true`, jsx syntax is supported, default `false`
// due to how esprima works, the AST is built when this option is `true`, even if
// `buildAST` is `false`
, jsx: true|false
// {Boolean}
// if true `result.code` is not assigned and therefore `undefined`
// if false (default) `result.code` property contains the result of `split.join`
nojoin: true|false
// {Object}
// overrides default parser `esprima-fb` and needs to be compatible with it
parser: require('esprima')
}
```
***return value***:
```js
{ ast
, tokens
, comments
, splits
, code
}
```
- ast `{Array}`: [abstract syntax tree](http://en.wikipedia.org/wiki/Abstract_syntax_tree) as returned by [esprima
parse](http://en.wikipedia.org/wiki/Abstract_syntax_tree)
- tokens `{Array}`: [tokens](http://en.wikipedia.org/wiki/Token_(parser)) provided by esprima (excluding
comments)
- comments `{Array}`: block and line comments as provided by esprima
- splits `{Array}`: code pieces split up, some of which where transformed as configured
- code `{String}`: transformed code, same as `splits.join('')` unless this step has been skipped (see opts)
## Browser Support
### AMD
Ensure to include [esprima](https://github.com/ariya/esprima) as one of your dependencies
```js
define(['redeyed'], function (redeyed) {
[ .. ]
});
```
### Attached to global window object
The `redeyed {Function}` will be exposed globally as `window.redeyed` - big surprise!
```html
<script type="text/javascript" src="https://unpkg.com/esprima"></script>
<script type="text/javascript" src="https://unpkg.com/redeyed"></script>
```
## redeyed in the wild
- [cardinal](https://github.com/thlorenz/cardinal): Syntax highlights JavaScript code with ANSI colors to be printed to
the terminal
- [peacock](http://thlorenz.github.com/peacock/): JavaScript syntax highlighter that generates html that is compatible
with pygments styles.
## Examples
- `npm explore redeyed; npm demo` will let you try the [browser example](https://github.com/thlorenz/redeyed/tree/master/examples/browser)
- `npm explore redeyed; npm demo-log` will let you try the [replace log example](https://github.com/thlorenz/redeyed/blob/master/examples/replace-log.js)
-140
View File
@@ -1,140 +0,0 @@
/*
* This config only contains ES5 tokens while the default config.js contains
* ES6 tokens as well.
*
* Copy this file and use it as a starting point for your redeyed config.
* Just fill in the tokens you want to surround/replace.
* Keep in mind that more specific configurations override less specific ones.
*/
module.exports = {
'Boolean': {
'true' : undefined
, 'false' : undefined
, _default : undefined
}
, 'Identifier': {
_default: undefined
}
, 'Null': {
_default: undefined
}
, 'Numeric': {
_default: undefined
}
, 'String': {
_default: undefined
}
, 'Keyword': {
'break' : undefined
, 'case' : undefined
, 'catch' : undefined
, 'continue' : undefined
, 'debugger' : undefined
, 'default' : undefined
, 'delete' : undefined
, 'do' : undefined
, 'else' : undefined
, 'finally' : undefined
, 'for' : undefined
, 'function' : undefined
, 'if' : undefined
, 'in' : undefined
, 'instanceof' : undefined
, 'new' : undefined
, 'return' : undefined
, 'switch' : undefined
, 'this' : undefined
, 'throw' : undefined
, 'try' : undefined
, 'typeof' : undefined
, 'var' : undefined
, 'void' : undefined
, 'while' : undefined
, 'with' : undefined
, _default : undefined
}
, 'Punctuator': {
';': undefined
, '.': undefined
, ',': undefined
, '{': undefined
, '}': undefined
, '(': undefined
, ')': undefined
, '[': undefined
, ']': undefined
, '<': undefined
, '>': undefined
, '+': undefined
, '-': undefined
, '*': undefined
, '%': undefined
, '&': undefined
, '|': undefined
, '^': undefined
, '!': undefined
, '~': undefined
, '?': undefined
, ':': undefined
, '=': undefined
, '<=': undefined
, '>=': undefined
, '==': undefined
, '!=': undefined
, '++': undefined
, '--': undefined
, '<<': undefined
, '>>': undefined
, '&&': undefined
, '||': undefined
, '+=': undefined
, '-=': undefined
, '*=': undefined
, '%=': undefined
, '&=': undefined
, '|=': undefined
, '^=': undefined
, '/=': undefined
, '===': undefined
, '!==': undefined
, '>>>': undefined
, '<<=': undefined
, '>>=': undefined
, '>>>=': undefined
, _default: undefined
}
// line comment
, Line: {
_default: undefined
}
/* block comment */
, Block: {
_default: undefined
}
, _default: undefined
}
-195
View File
@@ -1,195 +0,0 @@
/*
* Copy this file and use it as a starting point for your redeyed config.
* Just fill in the tokens you want to surround/replace.
* Keep in mind that more specific configurations override less specific ones.
*/
module.exports = {
'Boolean': {
'true' : undefined
, 'false' : undefined
, _default : undefined
}
, 'Identifier': {
_default: undefined
}
, 'Null': {
_default: undefined
}
, 'Numeric': {
_default: undefined
}
, 'String': {
_default: undefined
}
, 'Keyword': {
'break' : undefined
, 'case' : undefined
, 'catch' : undefined
, 'class' : undefined
, 'const' : undefined
, 'continue' : undefined
, 'debugger' : undefined
, 'default' : undefined
, 'delete' : undefined
, 'do' : undefined
, 'else' : undefined
, 'enum' : undefined
, 'export' : undefined
, 'extends' : undefined
, 'finally' : undefined
, 'for' : undefined
, 'function' : undefined
, 'if' : undefined
, 'implements' : undefined
, 'import' : undefined
, 'in' : undefined
, 'instanceof' : undefined
, 'interface' : undefined
, 'let' : undefined
, 'new' : undefined
, 'package' : undefined
, 'private' : undefined
, 'protected' : undefined
, 'public' : undefined
, 'return' : undefined
, 'static' : undefined
, 'super' : undefined
, 'switch' : undefined
, 'this' : undefined
, 'throw' : undefined
, 'try' : undefined
, 'typeof' : undefined
, 'var' : undefined
, 'void' : undefined
, 'while' : undefined
, 'with' : undefined
, 'yield' : undefined
, _default : undefined
}
, 'Punctuator': {
';': undefined
, '.': undefined
, ',': undefined
, '{': undefined
, '}': undefined
, '(': undefined
, ')': undefined
, '[': undefined
, ']': undefined
, '<': undefined
, '>': undefined
, '+': undefined
, '-': undefined
, '*': undefined
, '%': undefined
, '&': undefined
, '|': undefined
, '^': undefined
, '!': undefined
, '~': undefined
, '?': undefined
, ':': undefined
, '=': undefined
, '<=': undefined
, '>=': undefined
, '==': undefined
, '!=': undefined
, '++': undefined
, '--': undefined
, '<<': undefined
, '>>': undefined
, '&&': undefined
, '||': undefined
, '+=': undefined
, '-=': undefined
, '*=': undefined
, '%=': undefined
, '&=': undefined
, '|=': undefined
, '^=': undefined
, '/=': undefined
, '=>': undefined
, '**': undefined
, '===': undefined
, '!==': undefined
, '>>>': undefined
, '<<=': undefined
, '>>=': undefined
, '...': undefined
, '**=': undefined
, '>>>=': undefined
, _default: undefined
}
// line comment
, Line: {
_default: undefined
}
/* block comment */
, Block: {
_default: undefined
}
// JSX
, JSXAttribute: {
_default: undefined
}
, JSXClosingElement: {
_default: undefined
}
, JSXElement: {
_default: undefined
}
, JSXEmptyExpression: {
_default: undefined
}
, JSXExpressionContainer: {
_default: undefined
}
, JSXIdentifier: {
// many more identifies are possible, div, table, etc.
className: undefined
, _default: undefined
}
, JSXMemberExpression: {
_default: undefined
}
, JSXNamespacedName: {
_default: undefined
}
, JSXOpeningElement: {
_default: undefined
}
, JSXSpreadAttribute: {
_default: undefined
}
, JSXText: {
_default: undefined
}
, _default: undefined
}
-36
View File
@@ -1,36 +0,0 @@
{
"name": "redeyed",
"version": "2.1.1",
"description": "Takes JavaScript code, along with a config and returns the original code with tokens wrapped as configured.",
"author": "Thorsten Lorenz <thlorenz@gmx.de> (thlorenz.com)",
"main": "redeyed.js",
"scripts": {
"test": "npm run run-test && npm run lint",
"run-test": "tape test/*.js",
"lint": "standart",
"demo-log": "node examples/replace-log",
"demo": "cd examples/browser; open index.html"
},
"repository": {
"type": "git",
"url": "git://github.com/thlorenz/redeyed.git"
},
"keywords": [
"ast",
"syntax",
"tree",
"source",
"wrap",
"metadata"
],
"license": "MIT",
"devDependencies": {
"cardinal": "~1.0.0",
"readdirp": "~2.1.0",
"standart": "^6.1.0",
"tape": "~4.9.0"
},
"dependencies": {
"esprima": "~4.0.0"
}
}
-313
View File
@@ -1,313 +0,0 @@
;(function() {
'use strict'
/* global define */
var esprima
var exportFn
var toString = Object.prototype.toString
if (typeof module === 'object' && typeof module.exports === 'object' && typeof require === 'function') {
// server side
esprima = require('esprima')
exportFn = function(redeyed) { module.exports = redeyed }
bootstrap(esprima, exportFn)
} else if (typeof define === 'function' && define.amd) {
// client side
// amd
define(['esprima'], function(esprima) {
return bootstrap(esprima)
})
} else if (typeof window === 'object') {
// no amd -> attach to window if it exists
// Note that this requires 'esprima' to be defined on the window, so that script has to be loaded first
window.redeyed = bootstrap(window.esprima)
}
function bootstrap(esprima, exportFn) {
function isFunction(obj) {
return toString.call(obj) === '[object Function]'
}
function isString(obj) {
return toString.call(obj) === '[object String]'
}
function isObject(obj) {
return toString.call(obj) === '[object Object]'
}
function surroundWith(before, after) {
return function(s) { return before + s + after }
}
function isNonCircular(key) {
return key !== '_parent'
}
function objectizeString(value) {
var vals = value.split(':')
if (vals.length === 0 || vals.length > 2) {
throw new Error(
'illegal string config: ' + value +
'\nShould be of format "before:after"'
)
}
if (vals.length === 1 || vals[1].length === 0) {
return vals.indexOf(':') < 0 ? { _before: vals[0] } : { _after: vals[0] }
} else {
return { _before: vals[0], _after: vals[1] }
}
}
function objectize(node) {
// Converts 'bef:aft' to { _before: bef, _after: aft }
// and resolves undefined before/after from parent or root
function resolve(value, key) {
// resolve before/after from root or parent if it isn't present on the current node
if (!value._parent) return undefined
// Immediate parent
if (value._parent._default && value._parent._default[key]) return value._parent._default[key]
// Root
var root = value._parent._parent
if (!root) return undefined
return root._default ? root._default[key] : undefined
}
function process(key) {
var value = node[key]
if (!value) return
if (isFunction(value)) return
// normalize all strings to objects
if (isString(value)) {
node[key] = value = objectizeString(value)
}
value._parent = node
if (isObject(value)) {
if (!value._before && !value._after) return objectize(value)
// resolve missing _before or _after from parent(s)
// in case we only have either one on this node
value._before = value._before || resolve(value, '_before')
value._after = value._after || resolve(value, '_after')
return
}
throw new Error('nodes need to be either {String}, {Object} or {Function}.' + value + ' is neither.')
}
// Process _default ones first so children can resolve missing before/after from them
if (node._default) process('_default')
Object.keys(node)
.filter(function(key) {
return isNonCircular(key)
&& node.hasOwnProperty(key)
&& key !== '_before'
&& key !== '_after'
&& key !== '_default'
})
.forEach(process)
}
function functionize(node) {
Object.keys(node)
.filter(function(key) {
return isNonCircular(key) && node.hasOwnProperty(key)
})
.forEach(function(key) {
var value = node[key]
if (isFunction(value)) return
if (isObject(value)) {
if (!value._before && !value._after) return functionize(value)
// at this point before/after were "inherited" from the parent or root
// (see objectize)
var before = value._before || ''
var after = value._after || ''
node[key] = surroundWith(before, after)
return node[key]
}
})
}
function normalize(root) {
objectize(root)
functionize(root)
}
function mergeTokensAndComments(tokens, comments) {
var all = {}
function addToAllByRangeStart(t) { all[ t.range[0] ] = t }
tokens.forEach(addToAllByRangeStart)
comments.forEach(addToAllByRangeStart)
// keys are sorted automatically
return Object.keys(all)
.map(function(k) { return all[k] })
}
function redeyed(code, config, opts) {
opts = opts || {}
var parser = opts.parser || esprima
var jsx = !!opts.jsx
// tokenizer doesn't support JSX at this point (esprima@4.0.0)
// therefore we need to generate the AST via the parser not only to
// avoid the tokenizer from erroring but also to get JSXIdentifier tokens
var buildAst = jsx || !!opts.buildAst
var hashbang = ''
var ast
var tokens
var comments
var lastSplitEnd = 0
var splits = []
var transformedCode
var all
var info
// Replace hashbang line with empty whitespaces to preserve token locations
if (code[0] === '#' && code[1] === '!') {
hashbang = code.substr(0, code.indexOf('\n') + 1)
code = Array.apply(0, Array(hashbang.length)).join(' ') + '\n' + code.substr(hashbang.length)
}
if (buildAst) {
ast = parser.parse(code, { tokens: true, comment: true, range: true, loc: true, tolerant: true, jsx: true })
tokens = ast.tokens
comments = ast.comments
} else {
tokens = []
comments = []
parser.tokenize(code, { range: true, loc: true, comment: true }, function(token) {
if (token.type === 'LineComment') {
token.type = 'Line'
comments.push(token)
} else if (token.type === 'BlockComment') {
token.type = 'Block'
comments.push(token)
} else {
// Optimistically upgrade 'static' to a keyword
if (token.type === 'Identifier' && token.value === 'static') token.type = 'Keyword'
tokens.push(token)
}
})
}
normalize(config)
function tokenIndex(tokens, tkn, start) {
var current
var rangeStart = tkn.range[0]
for (current = start; current < tokens.length; current++) {
if (tokens[current].range[0] === rangeStart) return current
}
throw new Error('Token %s not found at or after index: %d', tkn, start)
}
function process(surround) {
var result
var currentIndex
var nextIndex
var skip = 0
var splitEnd
result = surround(code.slice(start, end), info)
if (isObject(result)) {
splits.push(result.replacement)
currentIndex = info.tokenIndex
nextIndex = tokenIndex(info.tokens, result.skipPastToken, currentIndex)
skip = nextIndex - currentIndex
splitEnd = skip > 0 ? tokens[nextIndex - 1].range[1] : end
} else {
splits.push(result)
splitEnd = end
}
return { skip: skip, splitEnd: splitEnd }
}
function addSplit(start, end, surround, info) {
var result
var skip = 0
if (start >= end) return
if (surround) {
result = process(surround)
skip = result.skip
lastSplitEnd = result.splitEnd
} else {
splits.push(code.slice(start, end))
lastSplitEnd = end
}
return skip
}
all = mergeTokensAndComments(tokens, comments)
for (var tokenIdx = 0; tokenIdx < all.length; tokenIdx++) {
var token = all[tokenIdx]
var surroundForType = config[token.type]
var surround
var start
var end
// At least the type (e.g., 'Keyword') needs to be specified for the token to be surrounded
if (surroundForType) {
// root defaults are only taken into account while resolving before/after otherwise
// a root default would apply to everything, even if no type default was specified
surround = surroundForType
&& surroundForType.hasOwnProperty(token.value)
&& surroundForType[token.value]
&& isFunction(surroundForType[token.value])
? surroundForType[token.value]
: surroundForType._default
start = token.range[0]
end = token.range[1]
addSplit(lastSplitEnd, start)
info = { tokenIndex: tokenIdx, tokens: all, ast: ast, code: code }
tokenIdx += addSplit(start, end, surround, info)
}
}
if (lastSplitEnd < code.length) {
addSplit(lastSplitEnd, code.length)
}
if (!opts.nojoin) {
transformedCode = splits.join('')
if (hashbang.length > 0) {
transformedCode = hashbang + transformedCode.substr(hashbang.length)
}
}
return {
ast : ast
, tokens : tokens
, comments : comments
, splits : splits
, code : transformedCode
}
}
return exportFn ? exportFn(redeyed) : redeyed
}
})()